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Foreword 


The second Transnational European Evaluation Project (TEEP II) was undertaken 
between August 2004 and June 2006. A methodology for evaluating transnational 
programmes had previously been tested during 2002-2003 by ENQA in the first 
Transnational European Evaluation Project (TEEP I). TEEP II was designed both to 
follow up and to develop further that work and so to continue to advance cross-border 
external quality assurance methods in an area where no single regulatory body can 
assume full responsibility. 

TEEP II had three primary objectives, to: 

• pilot the transnational quality evaluation of three different joint master’s 
programmes; 

• test the use of comparable quality evaluation criteria for joint master’s degrees; 

• develop a method for transnational external evaluation building on previous 
professional experience using common criteria. 

The project involved many actors across several countries: six quality assurance 
agencies, three programmes (each with partners in a minimum of five countries) and 
experts from across Europe. Each programme shared its ‘jointness’, but in other areas 
there was divergence: subject, philosophy, objectives, history, etc. This made it possible 
both to compare and to contrast the three programmes and so to draw conclusions from 
a more robust vantage point. 

The project was bracketed by two conferences, one at the beginning and one at the 
end of the exercise. Both conferences brought together all those involved and ensured 
a good level of mutual understanding with which to begin the project as well as an 
opportunity for shared reflection to close it. 

The present report emphasises the importance for joint programmes of developing 
quality assurance and enhancement processes which operate across the programme 
as a whole - wherever it is delivered. It also stresses the need to mirror the essential 
transnational character of the programmes in the external evaluation process - 
through the participation of experts and quality assurance organisations from different 
countries, working together. TEEP II has demonstrated successfully how that can be 
achieved. The report identifies a number of areas for further work and proposes that 
through working together with the European University Association (EUA) ENQA may 
be able to refine the methods for external and internal evaluation of joint master’s 
programmes. 

I should like to take this opportunity to thank all those who contributed to and 
therefore made possible this important project, including the representatives of the 
three programmes, the members of the expert panels and the management group 
which, under Staffan Wahlen’s able guidance, co-ordinated and ran the project. 

Peter Williams 

President, 

ENQA 


i. Executive summary 

The aim of TEEP II (the second Transnational European Evaluation Project) was to 
develop a method for evaluating joint master’s programmes. The project could also be 
seen as a means of engendering collaboration on a European scale. It involved the 
cooperation of six European quality assurance agencies and pilot evaluations of three 
different joint master’s programmes; each of which involved partners in at least five 
different countries. Overall, experts from nine different countries participated in the 
exercise. 

The criteria for the three pilot evaluations were based on the principles of the 
Standards and Guidelines for Quality Assurance in the European Higher Education 
Area. These were developed by ENQA in cooperation with ESIB, EUA and EURASHE 
in the context of the Bologna process, and were endorsed by the Ministers of Higher 
Education at the Bergen ministerial meeting of May 2005. However, the evaluations 
were specifically designed to test the special conditions which apply to reviews of joint 
programmes. These conditions included the development of criteria for assessment of 
organisation and management, content, and the programmes’ own quality assurance 
mechanisms under the specific circumstances. 

Taking these requirements into account, the criteria developed for the project (see 
Appendix II) refer primarily to the unity of the programme and progression between 
elements. Prerequisites, such as the organisation of student mobility, also had to be 
taken into account. Special attention had to be paid to processes of internal cooperation 
with regard to teaching, learning and assessment, to the dissemination of information 
and to staff development. The expectation of reasonable parity of outcomes also had 
to be considered. 

In order to be able to assess the programmes, the expert panellists involved were 
required to have a deep knowledge of the subject area(s) of the programmes under 
review and experience of transnational education and evaluation. In order fully to 
understand the relationships between the different parts of the programmes, it was 
desirable for them to visit as many partner institutions as possible. It was also 
important that the information from each site visit should be seen in the context of the 
programme as a whole and be forwarded to the main coordinator in order to contribute 
to its joint development. 

Regularly recurring accreditation and/or compulsory participation in national 
evaluation is a requirement for programmes in many European countries. Who, then, 
should be responsible for these joint master’s evaluations? TEEP II proposed that 
joint evaluations should be carried out by quality assurance organisations from two 
of the countries where the programme is in operation. It is suggested that the joint 
assessment made by these two organisations should be recognised as valid, after mutual 
agreement, for all the countries whose universities are involved in the programme. 

Such evaluations, including travel and fees paid to experts, it is proposed, should be 
funded through the quality assurance agencies in the countries where the programme 
is offered. 

In other types of evaluation (for development, or for international accreditation, 
labels, etc.) the programmes themselves would have to meet the costs. An estimate 
of such costs is given in chapter 5.2.6. 


A feedback conference, at which representatives of the programmes reviewed were 
given the opportunity of commenting on their experience of the pilot evaluations, 
concluded the project. Remarks were mainly positive. Programme reports, which also 
include responses from the programmes, are to be found on the ENQA website 1 . 


1 http://www.enqa.eu/pubs.lasso 


2. Introduction 


2.1 The report structure 

The report begins with a brief overview of the context and development of joint 
programmes and joint degrees. Chapter 3 describes the background to the project and 
the partners involved. In Chapter 4, the methodologies used in TEEP II are outlined 
and discussed, conclusions are drawn and lessons learned are identified. Chapter 5 
outlines a possible methodology for evaluating joint degree programmes in the future 
on the basis of the experience of the different participants in the project. The report 
ends with a summary of comments from the programmes and experts in Chapter 6 and 
some final observations in Chapter 7. 

2.2 The context of joint master's degrees 

2.2.1 THE BOLOGNA PROCESS, THE EUROPEAN COMMISSION AND 

ERASMUS MUNDUS, EUA AND ITS JOINT PROGRAMME PROJECTS 

International co-operation is an integral part of research and higher education. A range 
of exchange programmes have existed for a long time, and many young people have 
taken the opportunity to study abroad for a term or two. However, it is only in the last 
few decades that universities in different countries have begun to cooperate to develop 
programmes together in a systematic way. The ambition to develop European cross- 
border programmes (joint degree programmes), resulting in a truly trans-European 
degree, facilitating mobility and attracting students from outside the European Higher 
Education Area has existed for several years. It was mentioned in the Sorbonne 
Declaration (1998), and the Ministers of Higher Education involved in the Bologna 
process raised the issue at their meeting in Prague in 2001; it was an important item 
on their agenda at both Berlin (2003) and Bergen (2005) and will be followed up in the 
stocktaking process for the 2007 meeting in London. 

The European University Association (EUA) carried out a thorough survey of the 
existence and status of joint master’s degree programmes in Europe in 2002 2 and 
concluded that they “offer added value to European higher education”. The report 
recommended the further development of such programmes but pointed particularly 
to recognition and funding problems. In a more recent study 3 of 10 carefully selected 
networks three themes were highlighted: quality assurance and recognition; student 
experience and mobility; and curriculum integration and sustainability. The report 
stated that “institution(s) awarding the joint degree is/are responsible for quality - 
in line with the principles stipulated by their national systems” and that “partners are 
bound by jointly agreed requirements”. The project also contained useful advice for 
the establishment of new joint master’s programmes in the form of 10 “Golden Rules” 
(Appendix III). 


2 Survey on Masters Degrees and Joint Degrees in Europe by Christian Tauch and Andrejs Rauhvargers, September 2002: 
http://www.eua.be/eua/en/publications.jspx 

3 Results of the EUA Joint Masters Project on Developing Joint Masters Programmes for Europe, March 2002 - January 2004: 
http://www.eua.be/eua/en/publications.jspx 


As a result of recommendations in the above report a European Commission 
financed project on quality assurance of joint master’s programmes was initiated by 
EUA (European Master’s New Evaluation Methodology - EMNEM) parallel to TEEP 
II 4 . In contrast to TEEP II, however, the project dealt specifically with internal quality 
assurance. It is obvious that internal and external quality assurance are two sides 
of the same coin and that to some extent the same ground has been covered from 
two different perspectives. Hopefully, this combination will provide a more complete 
picture of the opportunities and challenges involved. 

In 2003 the European Union through its Commission initiated the Erasmus Mundus 
programme in order to boost joint European master’s programmes and make them 
attractive to students across the world. The Erasmus Mundus programme offers 
generous scholarships to students from third countries 5 as well as contributions towards 
costs involved in the administration of joint programmes. It is implemented in four 
actions: 

• Action 1 - Recognition of high-quality integrated courses at master’s level offered 
by a consortium of at least three universities in at least three different European 
countries. They must foresee a study period in at least two of the three universities 
and lead to the award of a recognised double, multiple or joint diploma. 

• Action 2 - A scholarship scheme for third-country graduate students and 
scholars from the whole world. This scholarship scheme addresses highly qualified 
individuals who come to Europe to follow the Erasmus Mundus Master’s Courses 
or to work for them. 

• Action 3 - Erasmus Mundus Master’s Courses selected under Action 1 also have 
the possibility of establishing partnerships with third-country higher education 
institutions. These partnerships allow for outgoing mobility of graduate EU 
students and scholars involved in the Erasmus Mundus Master’s Courses. 

• Action 4 - Enhancing attractiveness: It supports activities that improve the 
profile, the visibility and the accessibility of European higher education as well as 
issues crucial to the internationalisation of higher education, such as the mutual 
recognition of qualifications with third countries. 

TEEP II involved programmes selected in 2004 under Action 1. 6 

2.2.2 THE CONCEPT AND CURRENT STATUS OF JOINT DEGREES 

A student following a joint programme may end up with one or several types of 
diplomas. In most cases so far they have been awarded degrees issued by two 
universities (double degrees) or several universities (multiple degrees). In those 
cases successful students are provided with a qualification from each of the 
institutions involved. This arrangement does not, as a rule, present legal complications. 
Co-operation can be organised through agreements between partner universities and 
degrees can be awarded by each of the partners provided that they all have the 
competence and trust each other. In Finland the recommendation of the Ministry of 


4 As an outcome of the EMNEM project the EUA published Institutional Guidelines for Quality Enhancement of Joint Programmes, 
2006: http://www.eua.be/eua/en/publications.jspx 

5 A third country is defined by the European Commission as a country not included in the 25 EU Member States, the 3 EEA-EFTA 
states and the candidate countries for accession to the EU. 

6 Since 2004 the programmes have evolved to include the later Actions, e.g. to involve students from third countries. 


Education is that one of the members of a consortium should award the degree, but 
that it should be clearly indicated that it is offered in cooperation among several 
institutions. 

To establish truly joint degrees, however, presents legal difficulties with which 
a number of countries have struggled and some are still struggling. The 
Recommendation on the Recognition of Joint Degrees (Committee of the Convention 
on the Recognition of Qualifications, 2004) 7 defines the term in the following way: 

“A joint degree should, for the purposes of this Recommendation, be understood 
as referring to a higher education qualification issued jointly by at least two or 
more higher education institutions or jointly by one or more higher education 
institutions and other awarding bodies, on the basis of a study programme 
developed and/or provided jointly by the higher education institutions, possibly 
also in cooperation with other institutions. 

A joint degree may be issued as 

a. a joint diploma in addition to one or more national diplomas; 

b. a joint diploma issued by the institutions offering the study programme 
in question without being accompanied by any national diploma; 

c. one or more national diplomas issued officially as the only attestation of 
the joint qualification in question. 

The problem is that definition b. does not recognise the legality of the diploma. The 
definitions in the Recommendation are therefore not as widely accepted as they could 
have been. 

For our purposes a “genuine” joint degree may be defined as: a joint diploma 
issued by the institutions offering a joint programme in place of all the national 
diplomas, attesting the successful completion of this joint programme. For some, 
the question remains as to who is responsible for the award. This has been solved 
in different ways by different countries. However, in spite of the encouragement, and 
even strong recommendation, by the Ministers responsible for higher education in the 
countries involved in the Bologna process, the number of existing international joint 
degrees is still limited and legal problems still exist in several countries. 

TEEP II did not address the recognition challenges except in cases where they led 
to explicit difficulties for the further development of programmes. This applied, for 
example, to accreditation requirements in some countries. 

2.3 Introduction to TEEP II 

2.3.1 WHY TEEP II? 

As indicated above, the enthusiasm for developing programmes leading to joint degrees, 
especially master’s degrees, is shared by higher education institutions, Ministers of 
Higher Education in the European Higher Education Area (EHEA), and the European 
Commission alike. As a result, a growing number of programmes have been initiated 
in recent years. Many of these are attractive and highly successful. However, the 
transnational nature of the programmes presents particular challenges from both 
a practical, administrative point of view (e.g. the beginning and end of the academic 


7 http://www.coe. int/t/dg4/highereducation/recognition/lrc_EN.asp 


year, planning, information, student accommodation) and from a scholarly point of 
view (what is the nature of the often interdisciplinary research that the programme 
is based on?). 

Further, the pedagogical approaches in different universities in different countries 
need to be, if not harmonised, at least made explicit and related to the needs of the 
programme as a whole. Questions that need to be addressed include: What should be 
the extent of student activity in the learning and teaching process (oral and written 
presentations etc.)? How are the expected learning outcomes of a programme defined? 
Finally, the legal issue of national accreditation is a special problem that will have to be 
addressed in the further process of creating truly joint degrees. 

Currently, it is most often the prerogative of national governments to accept or 
reject a study programme as valid. In order to solve the problem of recognition/ 
accreditation of transnational evaluation it is essential to establish processes that do 
not involve unnecessary duplication, in the interest of both programmes involved and 
the quality assurance organisations, but also assure all those involved of the quality 
of the qualification awarded. The application of multiple national requirements is not 
sufficient for such purposes. External quality assurance of joint degrees will have to 
involve arrangements based on mutual understanding and trust. 

A methodology for evaluating transnational programmes was tested by ENQA in the 
first Transnational European Evaluation Project (TEEP I) 8 in 2002-2003. The report 
highlighted problems related to finding a common language and a common ground 
for the delivery of a programme but also the considerable benefits to the institutions 
and experts from many different countries involved in cooperating on a joint project. 
The project contributed to a “greater sharing of knowledge about approaches to 
quality assurance within European universities” 9 . The TEEP II project can be seen 
as a continuation and further development of cross-border external quality assurance 
methods in an area where no single regulatory body can assume full responsibility. 

The results could, to a large extent, be used to develop the evaluation of joint bachelor’s 
and doctoral degrees. 

It should be pointed out that, although there is broad consensus regarding the main 
processes of the methodology of evaluation of a higher education programme, the fact 
that a review may be carried out for different purposes must be taken into account. 
Should it be used ex ante for approval processes or is it rather intended for quality 
enhancement? The nature of criteria used for judgement or recommendations may 
vary with the aim, as will the general methodology, or at least the application of the 
methodology. These elements will depend on the answers to some of the fundamental 
questions of any evaluation: Why? For what purpose? For whom? 

This does not mean, however, that an evaluation carried out for quality assurance 
does not contribute to enhancement. The two purposes may interact, and a problem is 
rather that different quality assurance agencies in different countries may be invested 
with different powers as regards control. If, as we believe, the evaluation of joint 
programmes has to be carried out in cooperation among different agencies, it is 
essential that trust should be established and maintained among agencies, even if 
their approaches have different emphases. The experience of TEEP II showed that 


http://www.enqa. eu/projectitem.lasso?id=34835&cont=pastprojDetail 

9 TEEP I Methodological report, March 2004, p.10: http://www.enqa.eu/files/TEEPmethod.pdf. 


such cooperation may work very well when developing a model of evaluation for joint 
(master’s) degree programmes. The important thing is that the model used should make 
it possible for students, governments and other stakeholders in all the countries where 
the programmes are offered to rely on and have confidence in the quality of these 
programmes. 

2.3.2 BENEFITS 

For European higher education quality assurance agencies it is important to be able 
to use the results of the project in terms of identifying relevant criteria and a quality 
assurance methodology for joint master’s degree programmes. The nature of such 
programmes requires cooperation across borders. The way in which TEEP II was 
planned and implemented involved a considerable amount of collaboration between no 
less than six quality assurance agencies in different countries, which provided valuable 
experience for similar exercises in the future. 

The three programmes involved gained from the experience of external scrutiny 
of their activities, and other higher education institutions offering joint master’s 
programmes may be expected to benefit from the outcomes of the project by learning 
about the evaluation methods and criteria applied in the project. The previously 
mentioned EUA’s EMNEM project for developing principles for internal quality 
assurance of joint master’s programmes was significant in this context. Many of its 
findings were helpful to this project. 

The European Commission asked the TEEP II project to comment on three issues 
in particular: 

• development of a method of transnational evaluation financed on a feepaying basis 
by the university networks themselves; 

• development of a model of awarding a visible quality label for joint programmes; 

• use of culturally mixed teams of experts, not on perfect country-to-country match 
of partner universities and corresponding agencies. 

Reflections on these matters are of importance to the further progress of joint 
programmes and joint degrees, and are incorporated in the text in Chapters 4 and 5. 

An additional benefit of TEEP II was that it raised awareness of issues concerning 
joint degrees in a number of European countries. International co-operation in the 
project was considerable. Besides the six quality assurance agencies, the Management 
Group (MG; see Appendix I) established to run the project also included a project 
administrator from ENQA and representatives from EUA, the ENIC-NARIC network 
and the European Commission, one from each organisation. The panels included 
experts (including students) from a total of nine different countries, and the institutions 
involved in the project via the programmes were located across ten countries. 


3. Background and parties involved 

3.1 Overview of the evaluation model 

The general evaluation model used in the TEEP II project was based on the Standards 
and Guidelines for Quality Assurance in the European Higher Education Area 10 
developed by ENQA in cooperation with its European partners ESIB, EUA and 
EURASHE in the context of the Bologna process, and endorsed by the Ministers 
of Higher Education in the Bergen ministerial meeting of May 2005. 

The specific methodology as outlined in the application for the European 
Commission Socrates grant which funded the project involved the following: 

• testing of a common methodology and common predefined criteria; 

• self-evaluation exercise by each of the programme teams culminating in a self- 
evaluation report; 

• site visit by an international panel of experts (including both academic experts, 
quality assurance experts and a student) to discuss the self-evaluation report and 
gather additional information; 

• preparation of an evaluation report by each of the panels and feedback from each 
of the programme consortia; 

• preparation of a summary report on the methodology used and lessons learned. 

3.2 Organisation 

3.2.1 MANAGEMENT OF THE PROJECT 

The following six quality assurance agencies were selected by the ENQA Board to carry 
out the project: 

• Accreditation Organisation of the Netherlands and Flanders (NVAO); 

• Agency for Quality Assurance in the Catalan University System Higher Education 
(AQU), Catalunya; 

• Comite National d’Evaluation, (CNE), France; 

• Hungarian Accreditation Committee (HAC); 

• Quality Assurance Agency for Higher Education (QAA), UK; 

• Swedish National Agency for Higher Education (HSV), Sweden (Co-ordinator). 

It was deemed important that the project should be developed in close contact with 
other higher education organisations, particularly EUA and the ENIC-NARIC networks. 
Furthermore, it was natural that ENQA, as the initiator, should be involved in the 
running and administration of the project. 

As indicated earlier, the direct responsibility for the project was given to a 
management group (MG) with representatives of the above-mentioned organisations 
plus the European Commission. 

The MG met 10 times and had frequent email contacts throughout the two years 
of the project. 


lO 


http://www.enqa.eu/files/ENQA%20Bergen%20Report.pdf 


3.2.2 FOCUS OF THE PROJECT 

The focus of the evaluation was on the testing and development of methods and 
criteria for assuring the quality of joint master’s degree programmes. As it is clear 
from the programme reports, the project was concerned mainly with organisation 
and management and the programmes’ own quality assurance measures. Programme 
content in the sense of the literature used, the special orientation of courses or of the 
master’s theses was touched upon only to the extent that was necessary for the experts 
to understand the details of the programmes and was not referred to in the reports. In 
future evaluations these aspects will have to be considered. 

But content in the sense of, for example, the coherence of the programmes and the 
question of a common core was an important part of the evaluations and reports and 
is also referred to repeatedly in this report. 

3.2.3 PROCESS 

The project started in August 2004 with a meeting of the MG to prepare a detailed 
plan. The first step was the recruitment and selection of the programmes to be 
evaluated. It was already envisaged in the Socrates application that three programmes 
with five partners each would be involved, and it was agreed with the European 
Commission that they should be chosen from among the consortia that had 
been selected as Erasmus Mundus programmes. An invitation together with an 
accompanying letter from the Commission and a project plan was sent to all Erasmus 
Mundus beneficiaries in November 2004. 10 programmes expressed an interest in 
participating and three of them were selected by the MG. They were informed of their 
selection in January 2005 in a letter which invited them to a two-day launch conference 
in Stockholm in mid-March 2005. 

Simultaneously, the MG prepared a self-evaluation manual and guidelines for 
experts, and work started on recruiting subject experts for the evaluation panels, 
including master’s or doctoral students in the subject areas of the three programmes 
or related areas. The experts were appointed in February 2005 and invited to the launch 
seminar. 

The conference, which took place on 17-18 March 2005, provided an opportunity 
for representatives of the three programmes to meet with the MG and the experts 
and to agree on the basic principles of the project, on the final version of the self- 
evaluation manual and on the detailed planning of the process for each programme. 

The criteria were discussed generally and individually with each programme so as to 
make it possible to add such criteria as might be relevant for each of them, or exclude 
those that might not be pertinent. 

From March until the end of May 2005 the programmes conducted their self- 
evaluations. Self-evaluation reports were submitted (there was a certain amount 
of variation in the timing between the different programmes), and the site visits 
then commenced in June 2005 (in the case of one programme in September). The 
impressions from each site visit were reported back to the individual partner university 
in a short (3-5 pages) “feedback letter”. 

The site visits were concluded in November and programme reports were drafted by 
the three teams of quality assurance agencies responsible for each of the programmes. 
After approval by the experts, the reports were sent to the respective programmes for 
factual comment in January 2006. The programme reports were finally published on 


the ENQA website between July and August 2006 11 together with statements from the 
programmes. 

A methodological report was drafted between December 2005 and April 2006 and 
circulated to the experts and consortia for comments and discussion at the closing 
conference in early May 2006. Reflections and comments were incorporated in this 
final version of the methodological report. 

3.2.4 RECRUITMENT AND SELECTION OF PROGRAMMES 

As stated in 3.2.3, the programmes were chosen from among the 10 which had 
submitted their interest to the ENQA secretariat out of 19 selected for Erasmus Mundus 
status in 2004. The MG used the following criteria for selection: 

1. They should have been among the programmes selected for funding in the 
Erasmus Mundus framework; 

2. They should have produced graduates; 

3. For methodological and practical reasons, each degree consortium or network 
should have had no fewer than three and no more than five participating 
institutions; 

4. The programmes should have been offered in the countries of at least some of the 
agencies responsible for TEEP; 

5. Three different academic areas should have been represented: engineering (or 
other technical or science field), social sciences and humanities. 

6. The programmes could not expect to be funded for producing the self-evaluation 
report nor for extra meetings of their coordinating committees. 

It turned out to be somewhat difficult to recruit programmes that met all these 
requirements. The lack of resources to fund self-evaluations ruled out several potential 
programmes, and as a result, it was necessary to rank the criteria. Thus, criteria 2 and 3 
were ranked lower than the others. 

Three programmes were finally selected: 

• European Master of Arts in Media, Communication and Cultural Studies 
(CoMundus); 

• European Master in Law and Economics (EMLE); 

• Euro Hydro-Informatics and Water Management (EuroAquae). 

CoMundus and EMLE both had seven European partners, but five of those were 
selected for site visits on the basis of discussions between the agencies and the 
programmes. EuroAquae was newly developed and had not yet produced graduates. 

(See also 4.4.1) 

3.2.5 PROGRAMME DESCRIPTIONS 

The three programmes represent three different areas set out in the above mentioned 
criterion 5: humanities, social sciences, and engineering/science. They are different 
in a number of respects, some of which have an impact on the way they operate 
and, to a certain degree, on the way they were perceived by the expert panels. They 


ll 
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have different histories: two of them date back to the late nineteen-eighties and early 
nineteen-nineties, whereas one of them is new. They also differ in the number of 
partners in each consortium. 

CoMundus 

The CoMundus Programme is offered by a Consortium of seven European universities. 
The wider Consortium includes non-European partners who were not evaluated for the 
TEEP II project. The core of the Consortium was established, initially as a network, 
in 1988/89. 

The programme requires one and a half years of study, and awards 90 ECTS credit 
points including a thesis in the final semester. The core of the programme involves four 
major areas of study: 

• Theory, History, Epistemology of Media, Culture and Communications; 

• Institutions, Law, Politics, Economy of the Media; 

• Media as Text, Media Production and Analysis, Multimedia; and 

• Audience, Reception and Media Education. 

Each partner university must offer studies in each of the four areas for at least 10 ECTS 
credit points. Students apply to a home university, where they spend their first and 
third semesters and choose one host university (second semester) in one of the partner 
countries. The choice of the home and host university is based on the individual’s focus 
of studies as well as language ability. 

Student performance is assessed at the university at which the student is studying the 
given course, and the home university is responsible for marking the thesis. The final 
mark is based on the accumulated achievements. 

Quality assurance is the responsibility of each of the partner institutions. 

Since 1992/93 the programme has awarded a “European Master of Arts in Media, 
Communication and Cultural Studies”, which is issued as a double degree by the two 
universities the student has chosen to attend. 

EMLE 

The EMLE programme started in 1990 as the Erasmus Programme in Law and 
Economics at the universities of Rotterdam, Gent, Oxford and Paris IX. Since then, 
the number of partner universities has increased continually and the network now 
comprises ten teaching centres within the EU. Seven of them are part of the Erasmus 
Mundus programme. The programme also has links with universities in Israel and the 
United States. 

The programme covers one academic year (60 ECTS), subdivided into three terms, 
with courses starting in October and ending in June. In the first term, courses are 
offered at the universities of Rotterdam, Hamburg and Bologna while in the second 
term, courses are offered at Gent, Hamburg or Bologna. In the third term there is a 
range of courses and thesis topics to choose between, and courses are offered at the 
above-mentioned universities and in addition, at Aix-Marseille, Madrid, Manchester, 
Linkoping/Stockholm and Vienna. 

The programme is structured so as to give students basic courses in the first term, 
core courses covering the economic analysis of law in the second term and specialised 
courses in the third term. All courses are taught in English. Theses may also be written 


in the local language in the third term provided this is not the student’s mother tongue. 
There are plans to introduce common examinations and a joint grading system has been 
in existence from the beginning. 

Individual partner institutions are responsible for quality, and thus, the main 
responsibility for internal quality control lies with the local coordinators. However, 
a common course evaluation form has been developed and is used by all partners. 

On completion of the programme, students are awarded a double or multiple degree 
i.e. an official degree from each partner university where they have spent a term. They 
also receive an informal EMLE certificate. 

EuroAquae 

The University of Nice - Sophia Antipolis (F), Brandenburg Technical University 
Cottbus (DE), Budapest University of Technology and Economics (HU), Polytechnic 
University of Catalonia (ES) and University of Newcastle upon Tyne (UK) are the 
partners of the 4-semester master programme (2 years or 120 ECTS). Students have to 
study at a minimum of 3 locations. 

The Master is organised in a pedagogical continuum to provide introduction and 
common knowledge/soft skills (semester I - all locations); acquisition and the use 
of the hydroinformatics concepts, methods and tools (semester 2 - UK); a thematic 
specialisation: hydroinformatics systems, urban waters management, inland water 
management, decision support systems (semester 3 - all locations except UK); and for 
semester 4 (all locations), a research project or professional practice. 

The programme has set up an external quality monitoring group consisting of one 
academic expert and one external stakeholder representative. They have participated 
in the design of four cyclical quality assurance processes involving: 

• questionnaires to students, teaching staff, consortium member universities and 
future employers; 

• an evaluation of the published information about the programme; 

• an evaluation by the external experts of the questionnaire answers; 

• an evaluation of the professional and academic relevance of the programme. 

The rectors of the five universities have recently signed a document making joint 
awards in Hydroinformatics & Water Management possible from September 2006. 

Languages used in the programmes 

The language policies of EMLE and EuroAquae state that English is the universal 
language for teaching, teaching material, assessment and information. Other languages 
are permitted, for example for the final thesis, on certain conditions. In CoMundus, 
teaching is in the language of the country of each partner university, with the stated 
aim to ensure cultural diversity (the exception being the Danish partners, where 
teaching in English is widespread also for local students). The pros and cons of this 
policy are discussed at some length in the programme report. 

The opportunity to take courses in a second language (the languages of the countries 
of the partner universities) is offered, but the courses are not always well coordinated 
with the teaching of the programme specific courses. Furthermore, there is limited 
time for language study, especially in master’s programmes of 60 ECTS with the 
possibility of spending three months in three different countries. But longer stays do 


not necessarily lead to more contact with the native language of the country either in 
a formal learning situation or in day-to-day contacts with native speakers. 

3.2.6 RESOURCES (FOR MANAGEMENT OF PROJECT, FOR PROGRAMMES) 

The main source of revenue for TEEP II was a grant from the European Commission, 
which covered the travel costs of experts and MG members, meeting costs of the 
MG, the launching conference, the concluding conference, the reports and the 
administrative costs of the project. The participating agencies contributed through 
their work in the MG in meetings, the preparation of the self-evaluation manual and 
the guidelines for experts, the selection of experts, the running of the individual 
evaluations and the preparation of the programme reports and the methodological 
report. The experts (see Appendix 8.1) kindly agreed to participate pro bono. The three 
programmes produced self-evaluation reports, prepared the visits locally and organised 
and participated in the interviews with the visiting panels. Without all these generous 
contributions the project could not have been carried out. 

It is clear that in future evaluations, resources to cover the costs of the panel and 
various other expenses will have to be found. The question of who should foot that bill 
and how that may affect the implementation of the evaluation is discussed in section 
5.2.1 and cost estimates are given in section 5.2.6. 


4- Description and analysis 
of methodologies used 

TEEP II was a pilot project with certain natural constraints. For example, the 
responsibility for carrying out the evaluation was defined from the beginning. Similarly, 
resources were detailed in the project plan and allocated by the European Commission. 
The participating agencies, all members of ENQA, also contributed to the funding of 
the project through the work of the staff involved. Also the general framework of the 
evaluation model had been established from the outset. 

The main aim of the project was to develop a methodology for evaluating joint 
master’s programmes. As the project progressed, however, it became clear that in fact 
several methodologies had to be considered, mainly depending on whether the objective 
of the evaluation was quality enhancement or assurance. 

4.1 Selection and composition of expert panels 

4.1.1 DESCRIPTION 

The character of the TEEP II project required three kinds of experts on each of the 
panels: 

• academics, i.e. teachers and researchers in the fields under review; 

• quality assurance experts; 

• students. 

It was considered important that the panel members had to have as full an 
understanding of the nature of the joint programmes as possible. The academics had 
to have a thorough knowledge of at least part of the field covered by the programme 
and the capacity to see that field in relation to the programme as a whole. The 
quality assurance experts had to be familiar with current higher education quality 
assurance methodology and with recent developments of European cooperation in 
higher education. 

The members of the MG looked for experts for all the programmes in their respective 
countries in order to achieve diversity on all the panels. For practical reasons, the 
student members on the panels were recruited nationally by each of the quality 
assurance agencies represented in the MG. This turned out to be a workable solution. 

Names and CVs were supplied and further information was collected in order to have 
the best possible basis for selection. Finally the programmes were consulted in order 
to avoid any conflict of interest. One of the programmes emphasised the importance 
of avoiding conflicts of interest, particularly on the grounds that if a member of a 
panel was in the process of establishing a similar programme this might have created 
a problem of competition. 

The MG decided that the members from the six quality assurance agencies should 
join the panels as quality assurance experts. It was estimated that five panel members 
would be needed for each site visit. Past experience (e.g. the first TEEP project) had 
shown that it was unlikely that all panel members would have the opportunity to be 
present at all five site visits. It was thus considered necessary to establish a pool of 
a total of at least eight experts for each of the programmes to be evaluated. Ideally, 


these experts would have come from different European countries and preferably not 
from the countries of the institutions involved. The countries represented in the expert 
groups were as follows: 

CoMundus 

Agencies: HAC (Hungary) and QAA (UK) 

Academic experts: Sweden, Belgium, Ireland, Lithuania, UK 
Students: France, UK 

EMLE 

Agencies: HSV (Sweden) and NVAO (Flanders - the Netherlands) 

Academic experts: The Netherlands, Finland, Germany, Spain, UK 
Students: Sweden, Belgium 

EuroAquae 

Agencies: CNE (France), AQU (Spain) 

Academic experts: France, Hungary, Sweden, Spain, Spain 
Students: Hungary, Spain 

Each programme evaluation was coordinated by one of the agencies involved (the 
CoMundus evaluation was coordinated by HAC; EMLE by HSV; and EuroAquae by 
AQU), whereas the responsibility for organising each individual site visit and the 
secretarial functions alternated between the agencies of each agency pair. 

Although the composition of each visiting group varied to some extent, the 
organisation made it possible to have a reasonable degree of continuity across the site 
visits, which contributed to comparability and consistency of judgements. Two quality 
assurance experts from the two collaborating agencies were present at all visits, one 
of them functioning as organiser and secretary, the other focusing on the quality 
assurance aspects in the interviews. The student members (all except one of whom were 
PhD students in study areas related to the subjects of the programme) focused their 
questions on the experiences of students in the programmes. 

Certain differences in the way in which the teams have operated may be discerned: 

• The EMLE team did not appoint one person as chair but the chairmanship varied 
from site visit to site visit. The CoMundus team had two alternating chairs and 
EuroAquae three. 

• The EMLE panel was considered as one assessment panel. After each site visit the 
site panel shared their conclusions with the assessment panel as a whole in order 
for everyone to have a full overview of the programme. The CoMundus panel had 
a similar strategy. 

4.1.2 COMMENTS 

In the TEEP II process the agencies involved were responsible for the project from 
beginning to end. This included the selection and briefing of programmes and experts, 
the organisation of site visits and the writing of reports. In future evaluations this 
role will be assumed by one or several bodies external to the programmes reviewed. 

A discussion of what organisation(s) might be given that responsibility will be found 
in section 5.2.1. 


The procedure for the selection of TEEP II panel members involved scanning and 
searching on the basis of the MG’s familiarity with the field of higher education in 
their home countries and information from the evaluated programmes with regard 
to possible candidates. The outcome was successful, in that the experts functioned 
well as teams and gave valuable input to the evaluation process, analysis and 
recommendations. 

Finding subject area experts in specialised internationally oriented interdisciplinary 
areas may in future be more problematic. In several such programmes there may be 
a limited availability of experts, and the question of conflict of interest is therefore a 
delicate one. It is important, however, that content experts should be represented on the 
panels not least for the sake of the legitimacy of the outcome of the evaluation project. 
Students and other stakeholders have a very reasonable claim that the quality of the 
content should be addressed and evaluated. 

The composition of the teams in the TEEP II project corresponded well to the nature 
of the programmes. However, it should be pointed out that since all three programmes 
are mainly professionally, rather than theoretically, oriented, several of the discussions 
during the interviews at the site visits touched upon the professional relevance of the 
programmes. The question of employability was, therefore, addressed more in depth on 
those two panels which included a representative of a profession. 

The recruitment of student panel members may also be a delicate matter. Like the 
subject area experts, they must be familiar with the subject area under review. They 
must also have international experience and thus have an overall degree of legitimacy 
vis-a-vis the evaluated programmes. Suitable candidates maybe found via the National 
Unions of Students in Europe (ESIB) or through the national student unions, as was 
the case in the CoMundus evaluation. The other methods used in TEEP II, i.e. through 
MG members approaching universities in their respective countries or through personal 
contacts, were successful in this project, but working through established organisations 
may be advisable as a general methodological approach. 

In TEEP II the inclusion of students on the panels was a positive experience. They 
were all doctoral students who were conversant with (part of) the respective fields. 

They had had recent experience as students at master’s level and understood well the 
needs of students in an international environment. 

TEEP II used pools of experts. This required the exchange of knowledge and 
experiences, since not everybody could participate in all site visits. The MG’s experience 
in this respect was positive, and yet there were difficulties in making the whole panel 
think as one team. Ideally, therefore, the same knowledge and information should be 
shared by all simultaneously. This presupposes one group visiting all sites during a 
limited period of time. 

The two agency members in each of the panels played an active role in the 
discussions as organisers and quality assurance experts. Moreover, the programme 
secretary, who was one of the agency members, provided considerable input into the 
reports, although the reports represent the views of the team as a whole. 
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LESSONS LEARNED 

• It is advisable to limit the number of panel members. It may, however, be necessary to have a pool 
with more than one or two members of each category, from which to select participants for site visits. 
In that case, the transfer of knowledge and experiences is important; 

• A panel including subject area expertise, quality assurance expertise, students and, where relevant, 
stakeholders will bring together adequate expertise for a successful evaluation. Panel members should 
have international experience and an understanding of the character of joint provision; 

• Current students or recent graduates from master’s programmes similar to those under review will 
contribute to focusing relevant student-related questions in the evaluation process. 


4.2 Briefing of experts and programmes 

4.2.1 DESCRIPTION 

In order to launch the project, to agree on the basic principles, the general time 
frame and the planning of the individual evaluations a two-day meeting was held in 
Stockholm on March 17-18, 2005. The participants included representatives of the 
programmes to be reviewed, the expert panels and the members of the MG. 

The programme of the launch meeting (see Appendix IV) was devised so as to 
provide an opportunity for everybody involved to become familiar with the purpose 
and process of the project and to get to know each other. Thus there were four kinds 
of group meetings: 

• a plenary session at the beginning of the meeting, with an introduction to the 
project as a whole: aims, expected outcomes, process, reporting and timetable 
to be followed by a general discussion at the end to sum up the various points 
of view; 

• separate meetings of programmes to guide each of them through the self- 
evaluation document and process as well as the guidelines prepared for the 
experts, and to agree on any changes in the self-evaluation manual reflecting the 
specificities of each programme; 

• concurrent separate meetings of panels in order to agree on the roles and the 
tasks; and 

• joint meetings of each programme and “its” expert panel in order to agree on the 
parameters of each evaluation, dates of site visits, etc. 

4.2.2 COMMENTS 

While the above process was necessary for the particular purpose of TEEP II, it cannot 
be seen as a feature of a regular evaluation. 

The selected experts, though they may have experience in evaluation, as was the case 
in the TEEP II pools, will need advance briefing. The project funding did not allow 
for more than the brief introduction, at the launch meeting, to the specific nature of 
joint programmes and the rationale of the methodology provided. There is a need 
for a briefing session that explains the meaning of the process and the criteria and 
includes discussions on panel members’ views on their role as a team and as individual 
team members. The self-evaluation will have to be discussed and the extent to which it 
addresses strengths and weaknesses in relation to the criteria. 

It is also necessary to agree on the precise design of the evaluation, the timeframe, 
the exact format of the self-evaluation process and the timing and character of the 


site visits. This may be done through a meeting of a representative of the organisation 
responsible for review and the coordinator of the programme. 


LESSONS LEARNED 

• Briefing before the review and the preparation of a manual to support the panel will contribute to the 
relevance of the evaluation process; 

• An early meeting of the programme coordinator, a representative of the agency involved and the chair 
of the expert panel will lead to a common understanding of the evaluation process as a whole for all 
parties concerned. 


4.3 Self-evaluation 

4.3.1 DESCRIPTION 
The self-evaluation manual 

The aim of the self-evaluation exercise was for the programmes to reflect systematically 
on their own procedures and performance with regard to management and 
organisation, content (in the sense of coherence and relevance), and quality assurance. 

To support the self-evaluation exercise and to establish common ground, the MG 
prepared a manual (Appendix II) explaining the purpose and framework of the 
evaluation, and setting out the criteria to be applied in the self-evaluation and external 
evaluation processes. The manual further described the self-evaluation process and 
specified areas to be covered, as well as facts and figures relating to students, teaching 
resources, organisation and management and the programme’s own quality assurance 
processes which would form a key part of the evaluation’s evidence base. 

The self-evaluation process 

It was envisaged that each of the participating programmes would set up a group 
of representatives from all the partner institutions. The group would appoint a 
coordinating chairperson and include at least one student. It would get its legitimacy 
by being officially designated by the Board/Council/Committee responsible for the 
programme and circulate a draft version of its report among all the members of the 
consortium before finalisation. 

Ideally, such an exercise involves the participation of all the partners of the 
consortium, management, teaching staff and students. It addresses strengths as well 
as challenges with particular emphasis on the character of the joint programme. The 
outcome should be a self-evaluation report to be used as a frame of reference for the 
development of the programme itself and for the expert panel. The launch meeting was 
expected to provide the opportunity for programme-related modifications and additions 
to the self-evaluation manual. 

The three programmes went about the self-evaluation in slightly different ways: 

• CoMundus: The self-evaluation report was mainly prepared by a group of authors 
from the coordinating group. In dividing up the writing activities the group 
utilised the guidelines. The self-evaluation manual provided the basis for a two- 
day discussion by the whole Consortium at a meeting in London at the end of 
April 2005. 12 


12 


CoMundus Self-Evaluation Report, p. 22. 


• EMLE: The self-evaluation report was written by the coordinator and assistant 
coordinator. A draft version was sent to the coordinators of the partner 
universities in the Consortium and to some students of the programme as well, 
asking them for comments, which were then included in the document. 

• EuroAquae: The self-evaluation report was written by the coordinator. 

4.3.2 COMMENTS 
Self-evaluation process and reporting 

The contents of self-evaluations of joint master’s programmes differ from self-analyses 
of more traditional programmes in several ways. Questions to be discussed are, for 
example, the following 

a. The issue of the joint delivery must be addressed both from a content point 
of view (How are the aims of the programme met through the provision and 
different courses across the different institutions? How is progression assured 
through the programme?) and from a practical point of view (What are the 
arrangements to assist students with accommodation, visas, residence permits 
etc.?). An essential issue is the degree of centralisation versus institutional 
freedom, regarding both contents and delivery (teaching methods, assessment 
methods, student evaluation etc.). 

b. How is it ensured that the programme is at master’s level in relation to, for 
example, the Dublin descriptors? An answer to this question probably requires 
both consistent internal monitoring and follow-up of students. 

c. How do the constituent parts lead up to the aims of the programme as a whole, 
both academically and professionally? 

d. What are the relevant benchmarks and how have they contributed to the 
development of the programme? 

e. What is the added value of jointness? 

f. What are the quality assurance mechanisms for the programme, and how are 
they applied? 

g. What is the level of central support for the programme at the individual partner 
universities? 

In order to shed light on these issues, it is necessary to have the different partner 
universities’ views on how the above questions (and others, as developed in e.g. 
EMNEM 13 ) are addressed and resolved and on how each of them fulfils its own role. 
This does not necessarily mean that all partners must write their own reports. It may 
be reasonable for that to happen (EMNEM gives excellent recommendations for how 
it could be done), but for external evaluation purposes it is more essential that there is 
a common text agreed by all partners and to which they have given input, reflecting 
the challenges and strengths of the consortium as a whole as well as those of each 
individual partner. The CoMundus two-day consortium meeting (see above) is a fine 
example of how consensus can be reached, or different points of view highlighted. 

The input from students in the self-evaluation process is important. Collecting 
student views may be a particular problem, particularly in one-year master’s 




Institutional Guidelines for Quality Enhancement of Joint Programmes, EUA (see footnote 3). 


programmes, where they do not stay long in any one institution and none of them has 
experience from all partner institutions. Even if a student evaluation after each course 
is a regular feature in all partner institutions, students’ views on the programme as 
a whole may be difficult to obtain. The use of joint IT solutions such as electronic 
questionnaires or learning platforms such as Blackboard could solve many problems 
both in this respect and others. 

The three programmes involved in TEEP II had all developed routines for course 
evaluations, but none of them had implemented systematic models of either alumni or 
stakeholder surveys. With regard to the outcome of the (self-)evaluation, this may mean 
that there will be an emphasis on academic merit [in terms of the intrinsic values of 
the academic field(s) of which it is a part] rather than on usefulness to society at large. 
However, the monitoring model developed in EuroAquae, with two experts following 
the process continuously, is a useful way of obtaining relevant stakeholder input. 


LESSONS LEARNED 

• Regular reviews organised by the programmes themselves, as outlined in the European Standards 
and Guidelines, will serve as a useful tool for a successful self-evaluation process; 

• A systematic self-evaluation process which covers the programme as a whole and is based on reports 
or at least on substantial input from all individual partners contributes to the preparation of a report 
which is useful for the programme as well as for the visiting panel; 

• Addressing issues relating to the special characteristics of joint programmes helps to provide both the 
programme and the visiting team with a good understanding of these characteristics; 

• The implementation of a self-evaluation exercise leads to useful results if it functions as part of 

the programme’s own regular quality assurance procedures for collecting and analysing the views of 
students, alumni and stakeholders. 


4.4 Site visits 

4.4.1 DESCRIPTION 
Organisation of the site visits 

In all three programmes all partner universities were visited as planned. Two of the 
programmes have more partners than the five envisaged in the original application. In 
those cases site visits were carried out at the five partner institutions most relevant to 
the project, as agreed between the consortium and the panel. These were, for example, 
universities where compulsory courses were provided. If there were two partners in a 
country only one of them was visited, but representatives of both were present during 
the visit. 

The structure of the visits was similar across the three programmes. The schedule 
included a meeting of the panel the day before the actual interviews in order to agree 
on the questions to be put to the different groups and question strategies. The 
interviews were scheduled to last for about one hour with 15 minutes between every 
two groups to allow for follow-up discussions in the panel. A typical visit would be 
structured in the following way: 


Day 0 

Planning meeting of the evaluation panel 


Day 1 

09.00 

- 10.00 

10.00 

- 10.15 

10.15 

- 11.15 

11.15 

- 11.30 

11.30 

- 12.30 

12.30 

- 13.30 

13.30 

- 14.00 

14.00 

- 15.00 

15.00 

- 15.15 

15.15 

- 16.15 

16.15 

- 


Meeting with self-evaluation group/local coordinators 

Panel discussion 

Meeting with teaching staff 

Panel discussion 

Meeting with students (including alumni where possible) 

Lunch (panel only) 

Brief tour of learning resources, including library 

Meeting with rector and other senior management representatives 

Panel discussion 

Final discussion with self-evaluation group/local coordinator 
Final panel discussion 


As regards the different groups that the panels met, the institutions were asked to select 
5-10 people per group, but the outcome varied in size from category to category and 
from site to site and occasionally exceeded 10. The majority of groups tended to include 
between 5 and 7 members, and as was the case in TEEP I, the ambition was to avoid 
overlap of participants between interviews. This, however, could not always be avoided. 
In retrospect, interviewing some of the same persons from different angles sometimes 
proved a benefit. 

The order of the visits varied with each of the three programmes. In the EMLE 
evaluation, the coordinating institution was the first one visited. In the case of 
EuroAquae it was the last one, and with CoMundus it was in the middle of the process. 


4.4.2 COMMENTS 

Benefits and challenges of the site-visits 

Site visits by an external panel must be seen as an integral and indispensable part of the 
process. They are necessary for verification of statements made in the self-evaluation 
reports and to make sure that the programme functions as an integrated whole. Also, 
they provide an opportunity for programmes to discuss important issues with external 
experts in the field. 

There is no doubt that meeting representatives of all the partners of the consortia is 
an essential ingredient in the evaluation. In TEEP II the visits provided an opportunity 
to validate the information and impressions provided by the self-evaluation and to 
ascertain how, or even whether, partners met the criteria set out in the self-evaluation 
manual individually and as a consortium. They made it possible to form cumulative 
hypotheses regarding the programme as a whole by building on the results from one 
visit to the next. Interviewing students and staff from the different environments was 
important for the understanding of the functioning of the programme as a whole and 
of the relationship of the courses offered by the various partners of the consortium. 
Meeting senior management of the different institutions in person helped to draw 
conclusions as to their degree of commitment. 

In some cases, the panels of TEEP II met alumni of the programme, and one 
or two visits included interviews with employers/stakeholders. Both these groups 
contributed important information on employability and skills/learning outcomes 


related to working life. Yet, it is the experience of TEEP II that because of practical 
problems (e.g. a site may not be teaching in all semesters, or the visit may take 
place during an examination period) it may be difficult to find both the right kind of 
student/alumnus and the right kind of stakeholder. This raises the eternal question 
of the representativeness of interviewees, particularly of student groups. Additional 
ways of probing student and stakeholder opinion have to be found, e.g. internet-based 
questionnaires, the results of which could be discussed with management and teachers 
(see 4.3.2 above). 

The order of the visits may be significant for the result of the evaluation. The 
experience of TEEP II is that different arrangements may yield different results and 
that it is advisable to consider desired outcomes when planning the visits. Visiting the 
coordinating institution first may contribute to a good overview of the programme as a 
whole, and to formulating hypotheses about the programme which may or may not be 
verified at the subsequent sites. Visiting the coordinating institution last, may provide 
an opportunity to sum up and to clarify questions raised in the previous visits. 


LESSONS LEARNED 

• It is advantageous for an understanding of the programme to experience as many environments 
offered as possible; 

• An in-depth meeting of the panel before the first visit facilitates a common understanding of the 
programme as a whole and the development of hypotheses regarding the programme. It gives an 
opportunity to work out interviewing strategies and clarify the roles of each member during the 
interviews; 

• When planning the visits, the teaching schedules of sites should be taken into consideration to make 
sure all sites are active in the programme at the time of the visits; 

• The order in which the different institutions are visited may have an impact on the development of 
the interviews and the understanding of the programme; 

• The productiveness of the meetings with the different groups may be improved by the identification 
of areas for concern deriving from the self-evaluation report; 

• Interviewing both the programme management and institutional senior management, alumni and 
stakeholders contributes to a better understanding of the role of the programme at the individual 
institution; 

• Students who are nearing the end of a course or the programme have an overview which may yield 
useful information in interviews; 

• Where the timetable of the site visit is organised so as to give the expert panel the opportunity to 
discuss impressions between interviews and to sum up the main conclusions in a session at the end, 
it is easier to identify the main strengths and challenges. 


4.5 Reporting 

4.5.1 DESCRIPTION 

As stated above, the TEEP II project prepared three kinds of feedback for the 
programmes and as part of the project output. The immediate response was to the 
individual consortium partners visited. This was done in the form of a letter stating 
the panel’s views on the strengths and challenges of the particular site and its place 
in the programme. The second form of feedback was a report on the programme 
as a whole concentrating on administration and organisation as well as the quality 
assurance procedures in place (or not in place). The final feedback, the present 
methodological report, provides information on and analysis of the particular methods 


used in the TEEP II project and proposals for a methodology to be applied in the future 
accreditation and evaluation of joint degrees. 

Feedback letters 

The feedback letters were designed in essentially two distinct ways. The CoMundus 
feedback letters were organised in accordance with the three main themes of the 
evaluation: organisation and management, academic level and content and quality 
assurance. Each of these headings summarised positive features and development 
opportunities for the consortium as a whole and for the institution. The reports from 
the five sites built upon each other, in that the findings about the previous site or the 
whole programme were either corroborated or disproved with regard to their validity 
for the programme as a whole. 

In the EMLE evaluation, the letters focused on the institution visited and 
commented mainly on features of interest to quality assurance and the administration 
and organisation of teaching and student support, pointing out strengths and areas 
of concern. Remarks were made particularly with regard to the appropriate level of 
harmonisation of teaching, courses and assessment. 

EuroAquae feedback letters were similar to the EMLE letters. The EuroAquae 
evaluation panel felt, however, that the letters were not functional, since the main aim 
of the evaluation exercise was to report on the programme as a whole, and feedback to 
individual partners might contradict the conclusions drawn in the programme report. 

Programme reports 

The programme reports were designed according to a similar structure for which the 
MG prepared an outline with the main chapter headings. Partly to test and compare 
different reporting strategies, the MG decided not to provide an overly rigid framework, 
and consequently the actual reports differ to some extent in the application of the 
given structure. The common features involve an introductory chapter describing the 
TEEP II project, its methodologies and the programme under review, an evaluation 
of organisation and management, programme and programme delivery and quality 
assurance. The final chapter contains conclusions and recommendations to the 
programme as a whole. Although the criteria developed for the reviews are used and 
referred to consistently, the actual selection varies slightly between the reports. 

The CoMundus report identifies a number of recurring features affecting the quality 
of the programme, and comments specifically on good practice and developmental 
opportunities for each of the three sections (organisation and management, delivery 
and quality assurance). Here, the main chapter on the programme’s strengths 
and developmental opportunities also contains the panel’s recommendations. The 
concluding chapter is a brief summary. 

The EMLE report lists the main criteria for each of the main sections and discusses 
under headings including sets of criteria how and to what extent the programme meets 
them. The final chapter, which is comparatively short, summarises the observations and 
makes recommendations. 

The Euroaquae report is the longest of the three, and brings up points that are to be 
expected in a new programme, with more explicit references to the criteria than the 
other two reports. It discusses more in detail than the other two reports the problems 
of organisation and management, notably the question of student fees. It goes on to 


describe and analyse programme and programme delivery, particularly in terms of 
learning outcomes, and then brings up the particular challenges of quality assurance 
related to a new programme. Finally, there is a chapter on institutional issues, including 
a discussion of the kind of degree to be awarded and the question of equity between 
non-EU students and EU students. Description and analysis are interwoven throughout, 
except in the final conclusions and recommendations. 

The different structures of the programme reports are to a large extent a 
consequence of those of the feedback letters. 

4.5.2 COMMENTS 

The use and usefulness of two kinds of feedback 

The project required feedback to the individual partners visited as well as to the 
three programme coordinators. The letter to the partners was developed in order to 
provide relevant advice for the particular site and for the individual institution. The 
programme reports analysed the performance of both the programme as a whole and 
the constituent parts. In this way each institution got a reasonably full treatment in the 
context of the entire programme. In deciding on this scheme, the MG had in mind the 
nature of the TEEP II project as a pilot exercise, in which the voluntary cooperation 
and input on the part of the programmes deserved ongoing reflection, both for the MG 
and the programmes themselves. 

This procedure may, however, involve a risk of not addressing sufficiently the role 
of the site in the programme. Individual feedback may, therefore, affect the balance 
between the programme coordination and coordinator and the local coordinator. This 
balance may vary among programmes, and different strategies for reporting may have 
to be applied depending on whether, for example, there is strong programme leadership 
or more devolved programme management. The fundamental issue, which will have to 
be considered in reporting, is the added value provided by different kinds of feedback. 

It should be added that the feedback letters in TEEP II were considered useful by 
both the individual partner institutions and the programmes as a whole. 

The slightly different structures of the three programme reports did not seem to 
have any impact on reactions from the consortia, whose responses indicated that they 
were largely content with the discussions of strengths and challenges as well as with 
the recommendations. 


LESSONS LEARNED 

• The structure of reports may vary according to the nature of the programmes and the relationships 
between the partner institutions; 

• Whether written feedback should be given to the individual partner institutions may depend on the 
way in which the consortium as a whole is managed; 

• A report that discusses strengths and challenges with regard to the programme as a whole, and 
includes a list of panel recommendations is useful to programmes; 

• The language of the report should be clear and explicit, keeping in mind that not all readers of the 
report will be native speakers and may not understand the nuances of the report language. 


4.6 Criteria or reference-points? 

4.6.1 DESCRIPTION 

Having a shared understanding and use of particular terms is critical to the success 
of transnational quality assurance. Terms that are widely used but sometimes with 
different interpretations and applications include, for example, ‘criteria’, ‘standards’, 
‘guidelines’ and ‘reference points’. 

Agreement on the criteria and on their application was critical to the success of 
TEEP II. The term criterion is sometimes used for aspects that require a judgement 
on whether a particular level of performance has been met, or not. It is, however, 
also used for aspects that may be regarded as ‘reference points’ that invite reflection, 
discussion and commentary. The criteria applied in TEEP II were of both these kinds, 
and agreement on their individual use, and particular interpretation, was an important 
part of the discussions held at the opening conference and critical to the success of 
the project. 

The criteria used in TEEP II were developed from: 

• the criteria used in TEEP I; 

• the generic reference points for master’s degrees prepared by the Joint Quality 
Initiative (the so-called Dublin descriptors); 

• the “Golden Rules” for new joint master’s programmes established by the 
European University Association; and 

• the generic competencies developed within the TUNING project. 

The TEEP II criteria were broadly divided into three categories: organisation and 
management, programme content and delivery and quality assurance . 14 
The criteria regarding organisation and management define best practice in respect of 
aims, management, information and arrangements for welcoming and supporting guest 
students as well as policies concerning language. 

Criteria under the heading ‘Programme and Programme delivery’ include reference 
points for ensuring that the programme meets the requirements for the master’s degree. 
They also cover various generic competencies having to do with both the ability to work 
in an interdisciplinary and multicultural environment, and the capacity for synthesis 
and analysis, creativity and skills with regard to computing, decision-making and 
adapting to new situations. One of the most important items refers to the added value of 
programmes that are both interdisciplinary and transnational. 

Finally, the quality assurance criteria refer to joint strategies for quality assurance in 
order to ‘tease out’ the quality of jointness of a programme, and methods to ensure that 
aims are met and standards upheld. 

4.6.2 COMMENTS 

The criteria developed in the self-evaluation manual worked well for the purpose of 
determining the strength and sustainability of the programmes involved in TEEP II and 
the sustainability of the external evaluation process. The panels used statements from 
the self-evaluations as starting-points for explorative questions, and for judgements and 
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For a full list of all criteria, as well as references to TEEP I, the Dublin descriptors, the “Golden Rules” and the TUNING project, 
please see the self-evaluation manual (Appendix II). 


recommendations, explicitly in two of the programme reports, and more implicitly in 
one of them. 

The self-evaluation guidelines were a test vehicle for the quality assurance of the 
programme as a whole, including content and delivery. The evaluations included these 
aspects, mainly to make statements on coherence, comparability and sustainability 
across the programmes, but not on the individual courses. This is because the main 
focus of the project was not on contents but rather on organisation, management and 
internal quality assurance. Therefore, testing the validity of the criteria related to the 
competencies and learning outcomes (the TUNING statements and Dublin descriptors) 
proved to be a problem. In order to test in more detail than just the reference points 
for master’s level and generic competences, it would be necessary to delve more deeply 
into the quality of the contents of the programme, for example through reading theses 
or external examiners’ reports, than was possible in TEEP II. It is obvious, however, 
that defining and assessing the level of a programme and the standards of the degree is 
crucial for the success of the process. 

It is also clear that criteria related to contents of the programmes will have to be 
developed more in detail by the experts in collaboration with those under review. This 
is also foreseen in the proposed criteria, which assume that the programme should itself 
make appropriate analyses of learning outcomes, which will then serve as a basis for 
the evaluation by the expert panel. TEEP II involved programmes that demonstrate 
the considerable differences that exist among joint master’s level degree programmes 
in terms of length, coherence, number of partners and maturity. It is crucial that an 
evaluation should be able to demonstrate that the learning outcomes are compatible 
with the standards required for a master’s degree. 

The criteria (Golden Rules) developed by EUA 15 were especially useful for judging 
organisation and management, although they were designed particularly for the 
establishment of new joint programmes. A point that came out very clearly was the 
importance of institutional support for the smooth and effective operation and, in the 
last resort, survival of a joint master’s programme. 


LESSONS LEARNED 

• The criteria proposed in the Self-evaluation manual could serve as a basis for further development; 

• It should be made explicit how deeply the experts should probe the subject area and contents, e.g. 
the degree to which learning outcomes have been reached, the infrastructure such as the quality of 
laboratories and library holdings and the coherence of the curriculum; 

• Criteria regarding contents and delivery will have to be adapted to suit individual programmes as 
well as different purposes, but correspond to what may be required for a master’s degree. 
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4.7 A closing conference (and a follow-up?) 

In order to discuss the outcomes of the project, and particularly to confirm the lessons 
learned and the conclusions to be drawn by the different participants (Management 
Group, programme representatives and experts) a two-day conference was held in 
Stockholm on 4-5 May 2006. 

The programme of the closing conference was devised so as to provide an 
opportunity for all those involved to express their opinions on the process. Some of the 
conclusions are to be found in Chapter 6. 

It could be argued that future evaluations should have a similar sign-off session in 
order to discuss conclusions, recommendations and processes. This would give valuable 
information for further development. Such a seminar might include representatives 
from the different sites, programme management, student representatives, and, ideally, 
other stakeholders. 

It could, finally, also be maintained that, in the spirit of the European Standards 
and Guidelines for Quality Assurance, there should be follow-up procedures after a 
certain period of time. The Standards prescribe that a follow-up should take place in 
cases where an evaluation has contained recommendations for action or required an 
action plan. 

4.8 Role and cooperation of agencies 

4.8.1 DESCRIPTION 

TEEP II involved intense collaboration of six different quality assurance agencies from 
six different countries. The actual process of evaluating each of the programmes was 
conducted by two agencies working closely together to plan each of the stages from 
briefing programmes and experts to the writing of reports. Contacts were maintained 
via email and telephone, as well as through discussions at the meetings of the MG. The 
staff from the two agencies took turns organising site visits and taking responsibility 
for secretarial duties (cf. 3.2.1). 

The language used throughout was English. 

4.8.2 COMMENTS 

The arrangements for cooperation between all three pairs of agencies worked well and 
their relationships were productive and valuable. Joint efforts contributed to finding 
suitable experts from different countries. It was easy to agree on roles and the division 
of tasks. Communication via email and telephone was smooth, and there were few, if 
any, complications with regard to reaching agreement on the running of the projects 
and the conclusions to be drawn. 


LESSONS LEARNED 

• Cooperation between evaluation organisations works well if there is agreement on the division of roles 
and duties; there are enough resources and time allocated by each organisation. 


5. A proposal for a methodology 
for evaluating joint master’s degree 
programmes 

5.1 Points of departure 

5.1.1 FACTORS AFFECTING THE DESIGN AND STRUCTURE OF 
AN EVALUATION OF JOINT MASTER’S DEGREES 

The three master’s programmes involved in TEEP II were different with regard to age 
and maturity, the number of partners, coherence and length. These factors (and others) 
may have an impact on the design and structure of an evaluation. 

Age and maturity play a role in at least two ways: a programme which has been 
in existence for over a decade (as was the case with two of the TEEP programmes) 
has made it possible for the members of the consortium to get to know each other 
thoroughly. They have had the opportunity to try out different methods regarding 
organisation and management, teaching and assessment methods and quality assurance 
strategies. On the other hand, they may be losing the momentum for innovation and 
improvement. This may call for special focus on the sustainability of the programme. 

The number of partners involved will obviously play a role in the extent and cost of 
the evaluation and will have knock on effects for the size of the expert panel and the 
number of site visits, among other things. 

The coherence of the programme is dependent on how many courses of different 
kinds are involved, and how they relate to each other to constitute a whole. The 
TEEP II programmes differed in this respect. EMLE has essentially two components, 
which represent two quite different fields merging into a synthesis: the economic 
interpretation of law. EuroAquae aims to build up a programme with a wider choice of 
specialisation, in which the different parts build on each other, but still leave room for 
a fair amount of individual choice. CoMundus has four components, which represent 
somewhat different aspects of media and communication. Such differences will have 
to be taken into account, particularly in relation to the expertise needed for the panels. 

The relationship between ‘input’ factors such as curricular content, programme 
length etc and ‘output’ achievements, for example the learning outcomes, is a common 
theme in discussions about evaluation methodologies in higher education. The Dublin 
descriptors laid down in the Framework for Qualifications of the EHEA adopted by the 
ministers in their Berlin communique set out the learning outcomes associated with 
the award of master’s degrees. 

The master’s programmes evaluated in TEEP II differed in length from 6o ECTS 
(EMLE) to 120 ECTS (CoMundus). Whilst they were to some extent considered in 
relation to the Dublin descriptors, the project did not consider possible impacts of the 
different programme lengths on students’ academic achievements. However, from an 
organisational perspective, and with the importance attached to interviewing students, 
the arrangements for student mobility within the programmes did impact on the 
programme of site visits. 


5.1.2 DIFFERENT AUDIENCES HAVE DIFFERENT NEEDS AND REQUIRE 
DIFFERENT APPROACHES 

Depending on the purpose of the evaluation and the audiences addressed, the 
process and the outcome will differ. Ministries may need an assurance that the 
programme meets certain standards. Other stakeholders (employers, students) may 
require information on the quality of the programme in relation to that of other 
programmes. Evaluations undertaken to satisfy these audiences will be more oriented 
towards assessing the programme against predefined criteria. The programme itself, 
on the other hand, will need advice on how to enhance its quality. These two aims 
are not necessarily opposed. An assessment-oriented evaluation may very well provide 
programmes with ideas on how to improve. 

The evaluations of the three programmes involved in TEEP II had, to some 
extent, different emphases. The CoMundus review was more responsive to the specific 
expressed needs of the programme, whereas the other two were stricter with regard to 
the criteria and role played by the panel. In a sense, this reflects a difference (however 
small) between an enhancement-led exercise and an assessment-oriented one. 

Whichever the purpose, it is important that the evaluation gives the programme 
an opportunity for self-reflection and systematic self-evaluation of its own strengths 
and weaknesses, and that the report provides advice and recommendations for 
development. 

5.1.3 FITNESS FOR PURPOSE 

What is it that makes a joint master’s programme different from other programmes? 
The answer to this question, which has been touched upon before, has at least three 
dimensions: academic, relating to contents; pedagogical, relating to teaching and 
learning; and practical, relating to the conditions of students and teachers. Some of 
the challenges in relation to these dimensions are: 

1. The programme is provided as a single unified entity made up of components 
from different universities. The joint provision gives an added value, but also a 
challenge, compared to programmes taught by a single provider. This requires 
close cooperation to agree on purpose, contents and the curriculum as a whole, 
as well as the individual components taught at the partner institutions. This 
does not mean, for example, that only one teaching methodology or assessment 
method is applied, but that teachers and students must know what to expect and 
what is expected of them. 

2. Close cooperation is needed to agree on internal quality assurance procedures. 

3. From a practical point of view, students need support with everything from 
lodging, visas and residence permits, to social life in a new environment where 
the language barrier may be a problem. 

4. An information system which reaches out to all those involved (teachers, 
students, administrative staff and university management) is essential. 

5. Challenges related to external circumstances are national requirements 
for accreditation and differences in degree requirements between partner 
universities in different countries. 


The European standards and guidelines for external quality assurance 16 apply to the 
evaluation of joint degree programmes too, but the constraints listed above necessitate 
special considerations: 

1. The criteria for evaluation will have to address the above challenges. 

2. The purpose for which the evaluation is carried out must be clearly identified. 

3. The experts selected for the task of evaluating a joint master’s programme will 
have to have international experience and be able to appreciate the special 
conditions. 

4. The self-evaluation needs to reflect the views and roles of all the partner 
universities, as well as the challenges listed above. 

5. The question of who should be responsible for an evaluation of a joint master’s 
programme must be addressed and solved. 

With a view to the specific nature of joint master’s programmes and of the experiences 
gained by TEEP II, the MG proposes the evaluation model outlined in section 5.2. 

It is based on the assumption that it should accommodate the characteristics of the 
programmes and national requirements and also be an instrument for both quality 
enhancement and assurance. Further, it also considers that the costs incurred should be 
as limited as possible. This applies, regardless of whether the programme or a quality 
assurance organisation funds the external evaluation. 

5.1.4 PRINCIPLES 

The general principles of the Standards and Guidelines for Quality Assurance in the 
European Higher Education Area also apply to the evaluation of joint master’s degrees. 

Moreover, in order to obtain as full an understanding as possible of the quality 
of a programme it is advisable to assess both the prerequisites of the programme 
provision, the teaching and learning process and the outcomes. Prerequisites include, 
for example, organisation, funding and infrastructure. The process relates to teaching 
and assessment methods and internal quality management. Outcomes may refer to 
students’ learning outcomes and the extent to which graduates are satisfied with the 
programme in relation to their (future) employment. This structure will facilitate an 
understanding of the outcomes in the light of prerequisites and process. 

5.2 Proposal 

5.2.1 INITIATION AND GENERAL CONDITIONS 

Who takes the initiative and the responsibility for the evaluation of a joint programme 
will depend on the type and purpose of the review. In the view of the MG, different 
types of evaluations will require different methods. At the same time, the three variants 
outlined below in A-C are not, of course, mutually exclusive but contain common 
elements and can therefore be combined if appropriate: 

A. An evaluation of a joint programme is part of a mandatory accreditation/ 
evaluation process required by the legislation of one or several countries. 

In TEEP II, two of the programmes involved have undergone (partial) 
accreditation. EMLE has been accredited by NVAO, which accredits all provision 
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that leads to a Dutch or Flemish degree. In this particular case, the expert panel 
looked at the quality assurance arrangements, the outline of the programme 
and the qualifications of the teachers abroad. A few students and teachers from 
locations other than Flanders and the Netherlands were also interviewed. Site 
visits abroad were not undertaken, but may be part of the procedure, if necessary. 
As a consequence the programme was accredited as a Dutch degree. 

A similar exercise has been carried out by ZeVA (the Central Evaluation 
and Accreditation Agency, Hannover, Germany) with regard to the CoMundus 
programme. A site visit was coordinated with that of TEEP II. But the fact 
that the programme is accredited in one country does not automatically lead to 
accreditation in other countries. 

In the view of the MG the most natural way of solving the accreditation or 
evaluation problem is by way of mutual multilateral agreements among agencies 
provided they have an official mandate to do so. Such arrangements are discussed, 
planned, and, in some cases, already exist. Thus, for example, the European 
Consortium for Accreditation (ECA) 17 , created in 2003, has as its ultimate aim the 
achievement of mutual recognition of accreditation decisions among the partners 
before the end of 2007. 

The most far-reaching example of a successful implementation is the 
Washington Accord, an agreement among agencies in the U.S., Canada, Australia, 
Ireland, Hong Kong, Japan, New Zealand, South Africa and the United Kingdom 
in the area of engineering, which commits them to recognising each others’ 
accreditation systems. This arrangement makes it possible for graduates of a 
programme accredited by any quality assurance agency under the agreement 
to practice engineering in all areas covered by it. The Recommendation on 
the Recognition of Joint Degrees 18 by the Committee of the Convention on the 
Recognition of Qualifications (2004) proposes a similar arrangement for joint 
degrees. 

An agreement among European agencies with regard to joint (master’s) 
degree programmes would require only one evaluation/accreditation process for 
a programme to be recognised in all countries where a consortium operates. The 
process may be most usefully carried out by two accreditation/evaluation agencies 
in cooperation. The experience of TEEP II in this respect, as described in 4.8 
above, is positive. The model outlined in this report could, hopefully, be used as a 
point of departure for the methodology of such an undertaking. 

The choice of which agencies would undertake the evaluation would have, in 
each case, to be decided by agreement of the agencies of the countries in which 
the consortium is represented. We anticipate that the costs of the evaluation 
exercise would be shared among the respective agencies, since it is part of a 
procedure that is mandatory in the countries involved. 

It must be stressed that national legislation needs to be updated to 
accommodate acceptance of accreditation/evaluation by other countries (such 


17 http://www.ecaconsortium.net 

^ http://www.coe. int/t/dg4/highereducation/recognition/lrc_EN.asp 


legislation has already been introduced in Flanders). This was envisaged in the 
Bergen Communique, but until it becomes reality, case-by-case agreements will 
have to be worked out. 

B. An evaluation is part of a voluntary evaluation/accreditation process for the 
purpose of awarding a quality label. 

There is, it is argued, a need to highlight the very best of provision in European 
higher education both generally but also particularly to attract the best students 
in Europe and in third countries. One of the purposes of Erasmus Mundus is 
to promote and support such higher education around the world. A question 
may be asked about whether a ‘quality label’ would assist in the identification 
and promotion of such high quality programmes, providing a clear indication 
of excellence for prospective students from Europe and third countries and 
enhancing the attractiveness of such programmes. It might be reasonable to 
anticipate that programmes with a label could also serve as a benchmark for other 
joint master’s programmes. 

The idea of a European ‘quality label’ for joint master’s programmes raises 
several questions, however: who would benefit from a label? Who might/should 
award it, and on what basis? What responsibilities would it entail? 

It could be argued that the Erasmus Mundus status itself constitutes a form 
of quality label’. The selection process for Erasmus Mundus is, however, done 
solely as a desk exercise on the basis of written documentation provided with 
the application, including quality assurance aspects. There are not any visits to 
institutions or interviews with programme representatives or students, which 
could identify whether for example the intended outcomes are actually being 
met. A more interactive process would be needed for a substantive quality label’, 
although the precise aims of such a process would need to be clearly identified 
and a cost/benefit analysis undertaken to ensure that any methodology kept to 
the ‘necessary and sufficient’, thus providing the greatest chance of sustainable 
success. 

The general model for such a review may follow the principles outlined in 5.2.2 
- 5.2.5, but the criteria for assessment would have to be adapted to suit specific 
purposes. The programme, that would submit itself for review, might be expected 
to demonstrate consistent high quality in its organisation and management, and in 
the provision of teaching, and meet the high scholarly and professional standards 
required of a (master’s) degree. Crucially, it would have to have developed its 
own reliable and stringent quality assurance processes that reflected the ‘jointness’ 
of and across the programme, as opposed to there merely being actions that 
amounted to no more than a sum of (different) quality assurance processes in the 
different partners. 


It is unlikely that any label of value would be awarded unless a programme 
was well established and had produced graduates; feedback from alumni (and 
perhaps their employers as well) would be a significant element in any worthwhile 
evaluation. 

Who might run the evaluations and award the label? For an evaluation process 
to be credible it would have to be carried out by an organisation that had both the 
authority to award such a label and was acceptable at least to the higher education 
communities in the countries represented in the consortia, their ‘stakeholders’, 
and any local regulatory or evaluation bodies. 

With current uncertainties associated with wider mutual recognition of 
accreditation decisions made by national or regional bodies, it might be that a 
joint body would need to be created, perhaps from two or more quality assurance 
agencies relevant to the countries involved in any consortium of providers. It is 
unclear however on what basis such a body would have authority to award a 
label’, other than the intrinsic status of the organisations involved. What formal 
status might such a label have or need? Perhaps a label that has no formal (legal) 
status in itself, but is valued through the status of the organisations involved in its 
‘award’, would have market value. 

With increasing formal responsibilities associated with decision making 
processes of almost any kind, the award of such a label would require controls that 
would for example be able to demonstrate consistency of process, and there would 
be an inevitable question about whether some centralised process or monitoring 
might be needed. Who might do this, or have the authority to do it, is unclear 
at present. 

As in most other evaluations for the purpose of awarding a stamp of 
excellence (e.g. EQUIS, conducted by the European Foundation for Management 
Development), the costs will have to be borne by the programme, and be included 
in the budget. They will depend essentially on the number of members of the 
expert team and the number of site visits and meetings of the experts. 19 

The question was raised whether the evaluations conducted within TEEP II 
should have resulted in the awarding of labels. The MG concluded that this 
would have been entirely unsuitable, because TEEP II was a pilot project in which 
different methods were tried out. Further, the three programmes involved did not 
volunteer for that purpose. 

C. The evaluation is part of a voluntary process for the purpose of quality 
enhancement. 

An evaluation for the sole purpose of quality enhancement could, in principle, be 
carried out according to the model described in 5. 2-5.5. The exact implementation 
would, of course, depend on the specific needs of the programme. The difference 
between a solely enhancement oriented evaluation and other types would be that 
no judgement would be made about the programme’s meeting predefined criteria 
but that, rather, the focus would be on its strengths and weaknesses and on 
recommendations for enhancement. 
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For examples of cost estimates, see 5.2.6. 


5.2.2 SELECTION, COMPOSITION, BRIEFING AND WORKING METHODS 
OF EXTERNAL PANEL 

Standards and Guidelines for Quality Assurance in the European Higher Education 
Area state that external evaluation should include: 

• the use of international experts; 

• care in the selection of experts; 

• participation of students; 

• appropriate briefing or training for experts. 

The expert panels involved in the evaluation of joint programmes will need to include 
expertise in the subject areas of the programmes, which are often interdisciplinary 
and professionally oriented. This means that it is helpful if there is representation of 
subject experts in all fields covered by the programme, or, ideally, if there are experts 
who have relevant experience of interdisciplinary areas involved. It is also advisable, 
whenever relevant, to include stakeholders, e.g. possible employers with an interest in 
and knowledge of the area. 

In the case of reviews of joint programmes, ‘international’ also implies that 
experts should have considerable experience of transnational education and come from 
different countries, not necessarily those represented in the consortium. 

The experience of having students on the panel of experts in TEEP II was a positive 
one, and the MG proposes that this be the norm in evaluations of joint programmes. 

If the students have recently graduated from or are at the end of similar programmes, 
with knowledge of the advantages and challenges facing such studies, they will be in 
a position to make considerable contributions. 

It is vital that care should be exercised to avoid conflicts of interest. Such conflicts 
may arise from current or previous close contacts with one or several of the institutions 
under review, or plans to develop programmes similar to the one under review. 

Depending on the number of partners involved in a consortium, and on the number 
of sites to be visited, it may be necessary to appoint a pool of experts, from which to 
draw individuals to conduct the site visits. A site would normally be visited by a team 
consisting of 4-5 persons, including: 

• chair, who is also an academic expert; 

• academic expert and/or stakeholder representative; 

• student (alumnus); 

• secretary, who is also a quality assurance expert. 

As a general principle, it is advisable to keep the number of experts as low as possible, 
both to reduce costs and to establish a tightly-knit and efficient group. For tightly-knit 
programmes with few partners one expert could probably suffice, which would reduce 
the size of the visiting team to 4. 

Briefing experts initially and throughout is a necessary part of the process. In view 
of the costs involved in getting a panel together, some of the information will have to 
be available in a briefing package explaining the purpose, methods, roles and ethical 
considerations involved. E-mail would be a major means of communication, and the 
use of a joint digital project workspace would help to disseminate and update current 
information as well as to simplify the whole working process. 


In view of the TEEP II experience it is advisable to organise an initial meeting of 
the panel in order to reach agreement on purpose, process, interpretation of criteria 
and roles of panel members and to develop a common understanding of the programme 
under review. This may be done after the self-evaluation has been submitted and before 
or in conjunction with the first site visit. A final meeting after a draft version of the 
team’s report has been written could help to reach agreement on the conclusions to be 
drawn and the recommendations to be made. 

5.2.3 SELF-EVALUATION PROCESS 

The TEEP II Self-evaluation manual describes a process expected to “stimulate internal 
discussions within and among the participating institutions, include ample time and 
scope for reflection on emerging matters and provide opportunities to comment on and 
assist continuous improvement in the quality of the programmes.” 

In order to achieve this, the programme could most usefully utilise its “regular 
routines for collecting and analysing information about its own activities” 20 and, where 
necessary, supplement those with special investigations. It is essential that the process 
should take place with the involvement of all partner universities and include views of 
students, alumni and stakeholders. 

It is good practice to set up a group consisting of representatives from the 
different institutions, including students. For the coordination of the work it is to 
be recommended that a chairperson should be appointed, who is also the contact 
person with the external body responsible for the overall evaluation. The group and its 
chairperson should be officially designated by the leadership of the programme. 

The process and the report should take into account the criteria developed regarding 
organisation and management, programme level and content and quality assurance. 

The methodology and questions developed in EMNEM would be a valuable point of 
departure. 

5.2.4 SITE VISIT(S) 

A site visit (or visits) is an essential part of any evaluation and may be organised along 
different lines, depending on needs and resources: 

The panel chair may be given the task, together with a representative of the 
organisation responsible for the evaluation, of conducting a preliminary visit to the 
coordinating site to discuss any special requirements on the part of the panel or the 
programme and to prepare for the self-evaluation exercise and the evaluation process. 

With regard to the organisation of the visits, the whole panel needs to meet before 
the beginning of the process to discuss the programme and the self-evaluation. Such a 
meeting could be combined with the briefing session referred to in 5.2.2 or the first site 
visit. It is also essential that before each visit the panel should meet for a few hours to 
discuss the specifics and role of that site in relation to the programme as a whole. 

Alternative solutions for site visits may be chosen. One possibility is that all sites of 
the programme are visited and key groups are interviewed (programme coordinators, 
teachers, students, alumni, stakeholders, administrative staff institutional leadership). 
Each group should be interviewed separately in order for the views of the different 
categories to be heard. 


20 


Standards and Guidelines for Quality Assurance in the European Higher Education Area, 2005, p. 18. 


For reasons of cost, it may be necessary to reduce the number of visits. Other 
alternative solutions will then have to be found, which will ensure that the experts 
obtain sufficient information from relevant groups to be able to draw correct 
conclusions and give useful feedback. A minimum requirement is for the panel 
to visit the coordinating institution and for the other sites to be represented by 
students, teachers and management. The MG recommends the first version, however. 
Its experience with TEEP II was that almost every site visited highlighted different 
issues, which might have been overlooked in a single visit. It is difficult to appraise 
the different environments that students meet without firsthand knowledge, so visiting 
several (if not all) of the sites is beneficial. 

The order in which the sites are visited may vary with the strategy of the expert 
team. In TEEP II three different models were used (see 4.4.2), and the conclusion 
drawn was that visiting the coordinating institution first will help to formulate 
hypotheses of the functioning of the programme as a whole, whereas visiting the 
coordinating institution last will make it possible to draw conclusions based on the 
evidence from the previous visits. It is a question of a deductive vs. an inductive 
method. 

At the end of each site visit there should be enough time for the panel to sum up 
its discussions and draw preliminary conclusions. 

5.2.5 FEEDBACK AND REPORTING 

TEEP II produced two kinds of feedback: site reports (feedback letters) and final 
programme reports. As has been discussed above, there may not be a need for 
immediate feedback to each site, and it may even be counter-productive for the 
programme as a whole. 

The end product of the evaluation exercise is the panel’s report on the programme. 

It is good practice for the panel to submit a draft version to the programme coordinator 
for factual verification before publication. 

Reports may be structured in different ways depending on the needs of the 
programme under review. Considerations which require special attention may have 
emerged in the process. Reports resulting in accreditation may have to be structured 
differently from reports aiming at support and enhancement. In the MG’s view, 
however, a report should include the following main elements: 

• description of the programme; 

• assessment of organisation and management; 

• assessment of programme content and programme delivery, including reflections 
on the added value of ‘jointness’; 

• assessment of the programme’s own quality assurance processes; 

• features of good practice; 

• recommendations for improvement. 

Finally, if the budget allows, a feedback session could be held at the coordinating 
institution with the chair and secretary of the panel and representatives of the 
programme. 


5.2.6 ESTIMATED COSTS 

The costs involved in the evaluation of a joint degree programme will include 
expenditure for travel and per diem, fees for panel members and administrative costs. 
The estimates given below presuppose that panel members take part in the process as 
peers and colleagues on the basis of the principle of academic reciprocity. Consequently, 
the proposed fee is nominal. The estimates also take into account the fact that the 
report should be published on the Internet to avoid printing costs. They are further 
based on the actual costs of travel and allowances claimed by the experts in TEEP II 
in accordance with European Commission regulations. These amounted to 50 200€ for 
67 claims or 75o€ per claim. 

Two calculations are provided, which are based on different assumptions. One 
is a low-budget example, which involves a panel of four members visiting only the 
coordinating institution for two days, with representatives of the other partners of the 
consortium being present. The visit also includes a briefing session with the panel 
members. The other example assumes visits to five partner institutions by five panel 
members. The estimate of the number of working days is based on the number of 
visiting days plus extra days for preparation and reading report drafts. It is also assumed 
that the chair and the secretary will need extra days for the preparation of the report. 

The final costs of an evaluation will, of course, depend on the size of the expert 
panel, the number of partners and the number of sites visited as well as on the way in 
which various sessions can be combined. Different alternatives can be easily calculated 
on the basis of the examples below. It should be emphasised that neither example takes 
account of the costs of the consortium for partners’ travel arrangements or for the 
self-evaluation exercise including the writing of the report. 

TEEP II: Estimated costs 

Example 1. (a two-day site visit combined with a briefing session; 4 panel members) 


TYPE OF EXPENDITURE 

PERSONS 

DAYS 

AMOUNT 


Chair and secretary, 

10 days x 300 EUR x 2 persons 

2 

IO 

300€ 

6 ooo€ 

Other panel members, 

6 days x 200 EUR x 2 persons 

2 

6 

200€ 

2 400€ 

Travel costs 

Average travel costs 

4 


50o€ 

2 ooo€ 

Per diem (hotel and subsistence) 

4 

2 

250® 

2 ooo€ 

Secretarial work, administration 
and material 




1 ooo€ 


Total 13 400€ 


Example 2. (visits to five institutions, the first one combined with a briefing session; 
5 panel members) 


FEES FOR EXPERT PANEL MEMBERS 

PERSONS 

DAYS 

AMOUNT 


Chair and secretary 

2 

15 

30o€ 

9 ooo€ 

Other panel members 

3 

IO 

200€ 

6 ooo€ 

Travel costs 

Persons 

Sites/days 

Amount 


Average travel costs 

5 

5 

50o€ 

12 500c 

Per diem (hotel and subsistence) 

5 

5 

250c 

6 25o€€ 

Secretarial work, administration 
and material 




2 ooo€ 


Total 35 750€ 


5.2.7 A GENERIC MODEL AND EXAMPLE TIMESCALE OF AN EVALUATION 
PROCESS 


Identification of programme (s): through selection or submission 

2-3 months 

Clarification of purpose of evaluation: draft criteria established 

Selection of panel of experts 

Briefing of programme and experts and finalisation of criteria 

Programme (s) prepare self-evaluation documents 

3 months 

Site visits 

1-3 months 

Feedback and draft report 

Opportunity to comment on draft report 

1-2 months 

Final report published 

Possible feedback session with programme 

Possible follow-up, optimally after 2-3 years 


6. Feedback from programmes 
and closing conference 

The TEEP II project included one opening conference and one closing conference 
attended by the Management Group, the subject and student experts and 
representatives of the three programmes. The final conference was designed as a 
feedback mechanism, providing an opportunity for the programmes to exchange 
experiences and for the experts and agencies to share information on the reception 
of the evaluation outcomes. In this sense, it may be seen as an organic part of an 
evaluation process. In this case, for obvious reasons, the focus was on the methodology. 
Thus, one day was specifically devoted to group discussions on the methods applied in 
the project (see Appendix 8.4). Many of the comments have been incorporated into the 
main text. Some of the salient points raised in the discussion are found below. 

Focus of evaluation: Organisation - Content - Quality assurance 

As has been repeatedly stated in the previous chapters, the pilot evaluation focused 
on the organisation and management of the programmes and the adequacy of quality 
assurance arrangements. Content, mainly in the sense of coherence and progression 
was addressed more clearly in one of the programmes. It was generally agreed that in 
future evaluations questions of the quality of the provision would have to be assessed, 
regardless of the purpose of the evaluation (quality assurance or enhancement). This 
will require experts who are fully conversant with the different aspects of the field 
under review and have international experience. 

Joint vs. double or multiple degrees 

One of the programmes will shortly be awarding its first joint degrees, i.e. a document 
signed by the five universities involved, not accompanied by any national diploma. The 
other two issue double or multiple degrees. It was maintained that in certain cases 
students preferred a national degree, because it might be more prestigious than the joint 
degree awarded by a consortium, however well established this consortium might be. 
Nevertheless, this situation can be expected to change as joint degrees become more 
commonly accepted. 

Added value 

Evaluations would have to address the question of the added value of joint provision 
and joint degrees and assess the degree to which programmes offer value beyond that 
of a programme offered at a single university. Thus, the opportunities for international 
contacts, the value of international environments and different perspectives etc. will 
have to be assessed to a larger extent than has been the case in TEEP II. 

Different steps of the evaluation process 

The views of the experts and programmes largely mirror those expressed in this report. 
Thus there was general agreement that the presence of students and stakeholders on 
the panels was desirable. 


Site visits, preferably to all the sites, and interviews with all the partners represented 
in the programme, were considered to be essential in spite of the costs involved. It 
was suggested that costs would have to be covered through fundraising from external 
sources. In the long term, however, universities would have to include them in their 
budgets. 

The feedback letters to individual sites were found by at least one of the programmes 
to be of considerable value. They would, however, have to be communicated to the main 
coordinator of the programme in order to be of use in the planning and implementation 
of the programme as a whole and thus contribute to the joint development. 

Feedback letters from programmes 

In addition to the comments made at the final conference, the three programmes were 
given the opportunity to express their views on the programme reports and on the 
entire evaluation process in feedback letters to the MG. These formal responses from 
the programmes have been included as annexes in the programme reports. 

Generally speaking, the three feedback letters reflected points considered to be 
essential with regard to the programmes and the assessments. The EuroAquae response 
stressed three things, all related to the fact that they were a relatively new programme: 
their own internal formative evaluation, the fact that they would - from September 
2006 on - be awarding “genuine” joint degrees, and the fact that their graduates were 
joining major companies for professional practice during the fourth semester. These 
can all be considered part of successful development strategies in a recently started 
joint programme. 

CoMundus pointed to the current development strategies of the programme that 
had been established as a result of TEEP II. Among other things, they concerned 
strengthening the joint character of the programme by developing, for example, 
common research and publication activities through seminars, conferences and 
symposia. CoMundus was also looking for ways of institutionalising the programme in 
order to reach a stronger legal position as an organisational entity. 

The EMLE feedback letter was very helpful towards the finalisation of the 
programme report in question and of the methodological report. It was also concerned 
with factual clarifications and with the structure of part of the methodological report. 
The EMLE response pointed out that the greatest benefit of the TEEP II project had 
been to raise awareness across accreditation agencies with regard to the fact that joint 
master’s programmes needed joint quality assurance strategies. 


7. Concluding remarks and 
recommendations for further 
development 

7.1 What has the project shown? 

The aim of TEEP II was to develop a methodology for the evaluation of joint master’s 
programmes. If we mean by methodology a set of measures to be applied when 
evaluating the quality of the provision of transnational education at master’s level, 
this aim was achieved. The present report includes a proposal for how to initiate an 
evaluation under varying circumstances. The question of who should carry out an 
evaluation is discussed and a suggestion is put forward for mutual recognition by 
accreditation or evaluation agencies of evaluations of joint master’s programmes. In the 
report there is a model for a self-evaluation process, for selecting and briefing expert 
panels and for how site visits may be conducted. There is also a discussion on feedback 
and reporting. The self-evaluation manual (Appendix II) contains a proposed a set of 
criteria for assessment, which should, however, be modified to harmonise with the 
nature of the evaluation and the special character of the programme. 

The pilot evaluations indicated that the same overall methodology maybe applied for 
evaluations with different purposes and that the two main purposes, quality assurance 
and quality enhancement, can be the object of one and the same exercise and lead to 
tangible results in terms of effective measures taken by those under review. 

It may also be pointed out that the methodologies used in the pilot evaluations 
were reasonably successful in the way that they led to further development of the 
programmes under review. The extent to which this happened varied, but there is 
little doubt that the evaluations had an impact upon and contributed to changes in 
procedures and in some cases also in content. For example, one of the programmes 
was considering introducing a common core to be offered at all sites. 

7.1.1 THE CULTURE OF JOINTNESS 

The crucial object of evaluation is the jointness of the programmes. The question 
of a “culture of jointness” (an expression borrowed from the EMNEM report) is 
fundamental to the development of all joint programmes and to evaluations of such 
programmes. Any review must take this into account. It means that self-evaluations 
must address jointness both in the way they are carried out and in reporting. The 
selection and briefing of peers must reflect the character of this form of educational 
provision. Unfolding the common culture must be at the heart of the site visits, and 
this is one reason why as many sites as possible ought to be visited or at the very 
least represented at meetings. Evaluation reports must also take this into consideration. 
For example, the coordinators of a programme, and preferably all partners should be 
informed of any feedback to individual partner institutions. Finally, two or more quality 
assurance agencies working together in the organisation of the evaluation of joint 
programmes will add to the understanding of the unique character of transnational 
joint provision. 


7.1.2 LEGAL ASPECTS 

The legal aspects of joint versus double or multiple degrees were addressed to some 
extent in the pilot evaluations. They did not play a major role in the assessment 
exercise and different views on the role of the legal framework were expressed by 
the programmes involved. In the Management Group’s view, however, the provision 
in legislation for joint degrees in all the countries where such programmes are offered, 
would facilitate cooperation within the programmes as well as among quality assurance 
organisations. 

A related legal complication, which also affects reviews, is the definition of what 
constitutes a master’s degree. The three programmes involved in the pilot evaluations 
differed in length between 60 and 120 ECTS credits. Some countries define the 
minimum length as 60, others as 120. For evaluation purposes, this problem maybe 
solved by assessing a programme in terms of learning outcomes required by a master’s 
degree. The legislation in several countries, however, specifies the number of ECTS 
credits required for a master’s degree. In cases, involving countries with different legal 
requirements the evaluation of joint master’s programmes will be affected. 

7.2 Further developments 

Several references have been made in the previous chapters to the parallel project, 
EMNEM (European Masters New Evaluation Methodology), conducted by the 
European University Association. The aim of that project was to develop a methodology 
for internal quality assurance of joint master’s programmes within the university. The 
two projects, TEEP II and EMNEM, thus surveyed the same landscape from two 
different vantage points. For the sake of information-sharing EMNEM was represented 
in the TEEP II Management Group, and representatives of TEEP II participated in 
an EMNEM seminar and in the final meeting. Important insights were shared and 
duplication was avoided. 

The future development of joint master’s programmes and the evaluation of such 
provision will profit from further cooperation in this field. Thus a next step might be 
an evaluation of joint master’s programmes that would combine the methodology for 
external evaluation outlined in this report and the EMNEM Institutional Guidelines 
for Quality Enhancement in Joint Programmes. 
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Appendix II 

Self-evaluation manual 


Transnational European 
Evaluation Project II (TEEP II) 
Self-evaluation manual 


1. Background 

1.1 Aim of the project 

The current project forms the second stage of the ENQA Transnational European 
Evaluation Project (TEEP I) carried out in 2002-2003 1 . It aims to contribute to the 
development of a method for the evaluation of joint programmes and to the process 
of development of joint degrees in the European context. It does so by evaluating the 
organisation and management, level and content, and quality assurance systems of 
three European joint Masters programmes. 

1.2 Methodology 

The project is based on an internationally recognised evaluation model 2 and is 
undertaken in dialogue with higher education institutions. It involves: 

1. The testing of a common methodology and common criteria; 

2. The selection of three joint Masters programmes wishing to participate in the 
project; 

3. A self-evaluation exercise by each of the programme teams; 

4. The preparation of a self-evaluation report by each of the programme teams; 

5. A visit by an international panel of experts (including both subject area and 
quality assurance experts and a student) to discuss the self-evaluation report and 
gather additional information; 

6. The preparation of an evaluation report by each of the panels and feedback from 
each of the programme consortia; 

7. The preparation of a summary report on the methodology used and lessons 
learned. 

8. A contribution to the establishment of a methodology shared at the European 
level. 


1 http://www.enqa.eu 

2 The peer review model is a bottom-up approach to evaluation, practised by quality assurance agencies worldwide. Its benefits 
include providing legitimacy and encouraging self-reflection. Within the Bologna process, ENQA has a mandate to establish a 
peer review system for quality assurance agencies in Europe. 


1.3 Anticipated benefits for participating programme consortia 

The likely benefits from participating in the project include: 

• An opportunity to share experiences with other programmes, networks and peers, 
in order to assure continuous improvement of the programme quality and quality 
assurance; 

• The development of criteria that are commonly agreed and that have been tested 
and offer a dimension of transparency; 

• A contribution to the development of the quality assurance of joint degree 
programmes on the basis of the recommendations from the experts, and 
identification of good practice for comparable programmes and networks; 

• The opportunity to obtain feedback, which may help in identifying opportunities 
for improvement including the quality assurance ‘culture’; 

• The opportunity to promote the institutions, programmes and networks. 

2. Framework for evaluation 

This section sets out the framework for the evaluation of joint Masters programmes, 
including some criteria. In establishing the framework, the following have been 
considered: 

• The criteria used in TEEP I; 

• The generic reference points for Masters degrees suggested by the Joint Quality 
Initiative (so called Dublin descriptors); 

• The “Golden Rules” for new joint Masters programmes established by the 
European University Association; 

• The generic competencies developed within the TUNING project. 

Criteria and regulations that exist within national contexts are also taken into 
consideration. 

As the focus of the project is on the particularity of joint degrees at Masters level, the 
criteria have been adjusted to suit this purpose. Special account has also been taken of 
the interdisciplinary nature of the programmes. 

While content is an important aspect, the emphasis in the project is on the joint 
delivery of the programme and the quality assurance system attached to this. 

The criteria should not be interpreted as detailed prescriptions but rather as pointers 
towards the further development of programmes. The project is not intended to lead to 
either implicit or explicit ranking. 

2.1 Organisation and management 3 

Criteria under this heading could include: 

a. The aims of the programme are clearly defined. 

b. The processes of developing the aims and choosing partners for the programme 
should be interconnected. 


Section 2.1 draws upon the “Golden Rules” established by the EUA: http://www.eua.be/ 


c. The management of all participating institutions supports the goals and 
objectives of the programme. The programme is fully recognised by all 
participating institutions. 

d. Academic and administrative aspects of the programme are adequately staffed 
and funded. A sustainable funding strategy is in place. 

e. Mechanisms for co-operation, including degree of institutionalisation, role of 
each partner, financial management, communication system etc, are spelled out 
and understood by all parties. 

f. Responsibilities are clearly defined and shared amongst participating institutions. 
Lead roles and responsibilities are identified. 

g. Information about the programme is easily accessible to students and others. 

h. Arrangements for reaching out to and receiving guest students and scholars are 
in place, e.g. in terms of accommodation, mentor schemes, language courses, 
activities aiming at social integration, and assistance with visas and social 
insurance. 

i. The infrastructure, e.g. library and other information sources, premises and 
equipment, meets the needs of the programme. 

j. A language policy is in place. 

2.2 Level and content 

Criteria under this heading could include: 

a. The programme has established its own reference points, possibly utilising the 
Dublin descriptors 4 (below), to ensure that students achieve the competence level 
required for a Masters degree. 

Qualifications that signify completion of the second cycle are awarded 

to students who: 

• have demonstrated knowledge and understanding that is founded upon and 
extends and/or enhances that typically associated with Bachelors level, and that 
provides a basis or opportunity for originality in developing and/or applying 
ideas, often within a research context; 

• can apply their knowledge and understanding, and problem solving abilities in 
new or unfamiliar environments within broader (or multidisciplinary) contexts 
related to their field of study; 

• have the ability to integrate knowledge and handle complexity, and formulate 
judgements with incomplete or limited information, but that include reflecting 
on social and ethical responsibilities linked to the application of their knowledge 
and judgements; 

• can communicate their conclusions, and the knowledge and rationale 
underpinning these, to expert and non-expert audiences clearly and 
unambiguously; 

• have the learning skills to allow them to continue to study in a manner that may 
be largely self-directed or autonomous. 
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b. The programme encourages the further integrated development of generic 
competencies 5 such as: 

• Ability to work in an interdisciplinary team 

• Appreciation of diversity and multiculturality 

• Knowledge of the field of study 

• Knowledge of the profession 

• Capacity for analysis and synthesis 

• Capacity for applying knowledge in practice 

• Capacity for generating new ideas (creativity) 

• Capacity to adapt to new situations 

• Capacity to learn 

• Critical and self-critical abilities 

• Decision-making skills 

• Computing skills 

• Ethical commitment 

• Interpersonal skills 

• Knowledge of a second language 

• Research skills 

• Other 

c. The programme ensures the development of subject specific competencies. 

[NB. Subject specific criteria will be specified for each programme. The Stockholm 
launching conference provides an opportunity to discuss such criteria, and 
participants are encouraged to bring ideas and proposals to the conference.] 

d. The programme, through its joint delivery, provides an added value as compared 
to similar programmes delivered at national level. 

e. Teacher qualifications are sufficient and appropriate to the aims of the 
programme. 

f. Opportunities for staff development are provided. 

g. The programme is linked to research activities and/or recognised professional 
standards. 

h. The learning environment, including teaching and learning methods and 
assessment methods, favours the aims of the programme. Assessment methods 
are common to all parts of the programme or, at a minimum, agreed by all 
partner institutions. 

2.3 Quality assurance 

Criteria under this heading could include: 

a. The programme formulates and implements a joint quality assurance strategy/ies. 
Strategies may consider e.g. changes in student demand, external expectations, 
developments in teaching and learning, and new research areas. 

b. Quality assurance practices involve students, staff and other stakeholders from all 
participating institutions. 

c. The programme evaluates whether its aims are met and standards upheld. 


5 http://europa.eu.int/comm/education/policies/educ/tuning/tuning_en.html. While the TUNING statements are aimed at the 
first cycle level, they nonetheless provide useful reference points for joint Masters programmes too. 


d. Quality assurance includes coordination of assessment across the whole 
programme to ensure that all of its expected competences/learning outcomes are 
achieved. 

e. The programme develops mechanisms for follow-up and continuous 
improvement. 

3. The self-evaluation process 

This section presents practical information and advice on the self-evaluation exercise. 

3.1 Aim of the exercise 

It is important that the self-evaluation exercise fulfils the following aims: 

• to provide a framework to stimulate internal discussions within and among the 
participating institutions; 

• to include ample time and scope for reflection on emerging matters; 

• to provide opportunities to comment on and assist continuous improvement in the 
quality of the programmes. 

3.2 Self-evaluation group 

For the completion of each self-evaluation report, a group should be set up consisting 
of representatives from institutions participating in the programme. 

The group should appoint a chairperson, responsible for co-ordinating the work and 
for being the contact point between the self-evaluation group and the external body 
responsible for the overall evaluation. The group should include at least one student 
involved in the programme. 

The self-evaluation group (including the chairperson) should be officially designated 
by the Board/Council/Committee responsible for the programme. 

It is desirable that a draft of the self-evaluation report be circulated for comment 
among staff and students involved in the programme at each institution before it is 
submitted to the external body. 

Each self-evaluation group is responsible for the preparation of their report. 

3.3 Self-evaluation report 

The self-evaluation report should provide accurate documentation to be used by the 
panel of experts in their preparations, site visit, evaluation and report writing. It should 
also provide clear evidence of the reflective discussions that were an integral part of 
the self-evaluation exercise. 

The self-evaluation report should be not more than 25 pages in length, structured as 
far as possible in accordance with the headings of section 4 below. 

The report should be submitted electronically and in 6 paper copies to the contact 
person of the external body responsible for the evaluation of the programme. 


4- Themes for self-evaluation 

This section sets out the areas to be covered in the self-evaluation exercise and self- 
evaluation report. 

With the exception of the factual information requested in section 4.1, the report 
should express evaluation and analysis rather than sheer description. Identifying 
strengths, weaknesses, opportunities and threats will be an important aspect of this. 
The bulleted questions under each heading may serve as a starting point to your self- 
evaluation. They do not however constitute a compulsory or complete list of issues. 


4.1 Facts and figures 6 

4.1.1 BASELINE INFORMATION 

Name of programme 


Name of co-ordinating institution 


Names of other participating institutions 


Length of programme (academic years + ECTS) 


Website address of programme 



4.1.2 STUDENTS 

Please copy the table and provide the following information for each institution 
admitting students to the programme. 

INSTITUTION X 



No. of applicants 

No. of admissions 

No. of degrees 
awarded 

No. of students admitted 
from each institution 

w 

M 

w 

M 

w 

M 

X 

Y 

Z 


2002 











2003 











2004 












INSTITUTION Y 

etc... 


Facts and figures may be provided in a different format if more convenient. 


4.1.3 TEACHING RESOURCES 2004/2005 

Please copy the table and provide the following information for each institution 
contributing teaching resources to the programme. [NB. This table will be discussed 
at the Stockholm launching conference. Participants are encouraged to consider its 
appropriateness, and possible amendments.] 


INSTITUTION X 


Teacher 
(names not 
required) 

Title 

Contribution 
to the 

programme 

(full-time 

equivalent) 

Gender 

Academic 

degree 

Academic 

field/ 

discipline 

Professional 
qualification 
(if any) 

Research 
activity 
(member 
of research 
team) 

l 








2 








3 etc... 









INSTITUTION Y 

etc... 


4.2 Organisation and management 

4.2.1 AIMS OF THE PROGRAMME 

Please analyse and evaluate the aims of the programme, e.g. by addressing: 

• Why was the programme established? 

• What are the overall aims of the programme? 

• How were the aims determined? 

• Other 

4.2.2 CHOOSING PARTNERS FOR THE PROGRAMME 

Please analyse and evaluate the basis on which partner institutions were selected e.g. 
by addressing: 

• How/through what process did partner institutions come together to set up the 
programme? 

• What motivated the various partner institutions to join? 

• Other 

4.2.3 Mechanisms for co-operation 

Please analyse and evaluate the mechanisms of co-operation within the network, e.g. 
by addressing: 

• What are the respective roles of the programme co-ordinating team and the 
different institutions within the network? Are the roles clear and satisfactory? 

• To what extent have the mechanisms for co-operation been formalised and 
agreed? 

• How are financial matters managed within the network? Is the system 
satisfactory? 


• What are the methods of communication? 

• How often and under what circumstances does the programme co-ordinating 
team meet to discuss the running of the programme? 

• Do staff of the participating institutions meet to discuss their respective 
contributions to the programme? 

• Other 

4.2.4 SUPPORT AND RECOGNITION 

Please analyse and evaluate the level of support and recognition that the programme 
receives, e.g. by addressing: 

• What is the legal status of the programme and the award? 

• What are the respective roles of the different institutions in the recognition 
process? 

• Have you encountered any obstacles with regard to recognition? If so, what? 

• In what way is the programme supported, e.g. by management of the individual 
institutions? 

• Other 

4.2.5 STAFFING AND FUNDING 

Please analyse and evaluate the staffing and funding of the programme, e.g. by 
addressing: 

• Is there a joint funding strategy for the programme? If so, please describe this. 

• On what basis are joint resources distributed? 

• What academic and administrative staff resources are attached to the programme? 
Please comment on teaching resource figures provided at 4.1.3. 

• What funding requirements does the running of a joint programme have 
compared to a programme provided by one institution only? 

• Generally speaking, are the resources available to the programme sufficient? 

• Other 

4.2.6 INFRASTRUCTURE 

Please analyse and evaluate the infrastructure available to the programme at each of 
the partner institutions, e.g. in terms of IT equipment, library and information services, 
and premises at the various partner institutions. 

4.2.7 RECRUITMENT AND INFORMATION 

Please analyse and evaluate the student recruitment system, e.g. by addressing: 

• How are prospective students informed about the programme by the various 
partner institutions? 

• Other 

4.2.8 RECEPTION OF STUDENTS AND SCHOLARS 

Please analyse and evaluate the arrangements for receiving guest students and scholars, 
e.g. by addressing: 

• How are students introduced to the programme and to the courses offered at each 
institution? 


• What kind of assistance do guest students and scholars get at the various 
institutions? Is the support system satisfactory? 

• Does the cultural heterogeneity of the student group require special attention? If 
so, in what way? How may the cultural heterogeneity benefit the programme? 

• Other 

4.2.9 LANGUAGE POLICY 

Please analyse and evaluate the language policy of the programme, e.g. by addressing: 

• What are the language requirements for completing the programme? 

• What special provisions are made regarding language? 

• Does language pose any particular problems? If so, how are they solved? 

• Other 

4.3 Level and content 

4.3.1 PROGRAMME STRUCTURE 

Please analyse and evaluate the structure of the programme, e.g. by addressing: 

• To what extent does the programme have a compulsory ‘core’? 

• What are the optional elements? 

• What is the progression within the programme? 

• Other 

Please enclose a programme description, e.g. in the form of a chart, outlining the 
different parts of the programme including compulsory and optional elements. 

4.3.2 ACADEMIC STANDARD 

Please analyse and evaluate the academic standard of the programme, e.g. by 
addressing: 

• How is compatibility of academic standards throughout the programme achieved? 

• Does the interdisciplinary nature of the programme present particular challenges? 

• What academic added value is achieved through the joint delivery of the 
programme? 

• What are the student admission requirements? What is the academic level of 
students at entry? 

• To what extent does the profile of the academic staff match the aims of the 
programme? 

• What are the opportunities for staff development? 

• In what way is the programme linked to research and professional activities at the 
participating institutions? 

• Other 

Please enclose reading lists for the various parts of the programme. 

4.3.3. TEACHING AND LEARNING METHODS 

Please analyse and evaluate the teaching and learning methods used in the programme, 
e.g. by addressing: 

• What teaching and learning methods are used within the programme (e.g. 
lectures, small group teaching, seminars, course work, thesis preparation/ 
supervision, laboratory work, trainee placements etc.)? What methods are the 
most commonly used? What are their respective pros and cons? 


• What assessment methods are used within the programme (e.g. written 
examination, laboratory experiment write-up, essay, oral examination, project 
report)? What methods are most commonly used? What are their respective pros 
and cons? 

• Is there a common approach to teaching and learning methods within the 
programme or do these vary from one institution to another, and/or between 
different stages in the programme? If so, how? 

• In what way does the learning environment favour academic achievement? 

• To what extent do the teaching and learning methods contribute to critical and 
independent thinking? 

• Other 

4.3.4 COMPETENCES/LEARNING OUTCOMES 

Please analyse and evaluate the expected competences/intended learning outcomes 

of the programme, e.g. by addressing: 

• What are the expected competences/intended learning outcomes of the 
programme? Which of these are subject-related and generic, respectively? 

• How were the expected competences/intended learning outcomes developed? 

• Are the expected competences/intended learning outcomes published and made 
available to students and staff? 

• To what extent are the expected competences/intended learning outcomes 
designed to reflect the aims of the programme? 

• Are the “Dublin descriptors” and/or the competencies developed within the 
TUNING project relevant to the programme? 

• To what extent have the expected competences/intended learning outcomes been 
attained? How do you know? 

• Other 

4.4 Quality assurance 

4.4.1 QUALITY ASSURANCE STRATEGIES 

Please analyse and evaluate your strategic work on quality assurance, e.g. by addressing: 

• What is the overall philosophy and strategy for quality assurance at programme 
level? 

• Do separate strategies exist for quality assurance at programme and thematic/ 
course level? 

• Other 

Please enclose strategic document/s, if available. 

4.4.2 PARTICIPATION IN QUALITY ASSURANCE 

Please analyse and evaluate the level of participation in quality assurance, e.g. by 

addressing: 

• Where does the responsibility for quality assurance at programme level lie? 

• What parties are otherwise involved (e.g. at thematic/course level)? 

• What is the level of participation by students, staff and/or other stakeholders? 

• Other 


4.4.3 QUALITY ASSURANCE PRACTICES 

Please analyse and evaluate your quality assurance practices, e.g. by addressing: 

• How is quality assurance organised at programme level and thematic/course level, 
respectively? 

• How is the programme evaluated (e.g. with regard to aims and objectives, student 
competences/learning outcomes, teacher competence and staff development, 
teaching and learning methods, relevance in relation to labour market 
requirements)? 

• What use is made of input from e.g. reports from accrediting or other 
external bodies, staff and student feedback, feedback from former students 
and their employers, response from professional organisations, labour market 
representatives or discipline associations, student progress information, external 
examiners’ reports? 

• Other 

4.4.4 FOLLOW-UP AND IMPROVEMENT 

Please analyse and evaluate the mechanisms for follow-up and continuous 
improvement, e.g. by addressing: 

• Where does responsibility for follow-up lie? 

• How are shortcomings identified and remedied? 

• Other 

Please provide example/s, if any, of how the quality assurance practices have resulted in 
changes to the programme to the benefit of staff, students and/or other stakeholders. 

4.4.5 THE SELF-EVALUATION EXERCISE 

If you wish, please comment on your experience from this self-evaluation exercise, e.g. 
by addressing: 

• What is the usefulness of the guidelines? 

• What alterations do you suggest, if any? 

• Other 

Your feedback is most valuable in the process of developing a model for the evaluation 
of joint degrees. 

4.5 Summary 

Please provide a summary of the main strengths of the programme and the main 
challenges it is facing, as you see it. 

Checklist 

Please submit the following documentation to the contact person of the external body 
responsible for the evaluation of the programme: 

Q Self-evaluation report (£ 25 pages) 

□ Quality assurance strategy/ies (if available) 

G Programme description (e.g. chart) outlining the different parts of the programme 
including compulsory and optional elements 

□ Reading lists for the various parts of the programme 


The documentation should be submitted electronically as well as in 6 paper copies. 


Appendix III 

EUA Golden Rules for New Joint Masters Programmes 


1. Know why your are setting up the programme 

2. Choose your partners carefully 

3. Develop well-defined programme goals and student-learning outcomes with your 
network partners 

4. Make sure that all the institutions (and not just academic colleagues) fully 
support the goals and objectives of the programme 

5. Ensure that sufficient academic and administrative staff resources are involved 
in the programme 

6. Ensure that a sustainable funding strategy for the programme is in place 

7. Take care that information about the programme is easily accessible to students 

8. Organise and plan sufficient meetings in advance 

9. Develop language policy and encourage local language learning 

10. Decide who is responsible for what 
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Appendix IV 

Launch conference programme 

Agenda for the TEEP II Launching Conference 

Conference venue: Hotel SAS Radisson Royal Park Hotel, Frosundaviks Alle 15, Solna, 
tel. +46 (0)8 624 55 00 

THURSDAY 17 MARCH 


TIME 

ACTIVITY 

Morning 

Arrival 

12.00 - 13.00 

Lunch 

13.00 - 13.15 

Welcome and introduction 

• Sigbrit Franke, University Chancellor, National Agency for Higher Education 
(HSV), Sweden 
Chair: Staffan Wahlen (HSV) 

13.15 - 14.15 

Short introduction of participating programmes and agencies 1 
Chair: Staffan Wahlen (HSV) 

14.15 - 15.15 

Presentation of TEEP II: 

• European Association for Quality Assurance in Higher Education (ENQA) and 
TEEP II, Emmi Helle, ENQA secretariat 

• Experience from TEEP I, Nick Harris, Quality Assurance Agency for Higher 
Education (QAA), UK 

• Introduction to the project, Staffan Wahlen (HSV) 

• Discussion 

15-15 - 15-45 

Coffee 

15-45 - 17-0° 

Self-evaluation process 

• Fiona Crozier (QAA) 

• Discussion 

17:00 - 18:00 

CoMundus group work: 

• Project planning 

Chairs: Christina Rozsnyai, Hungarian Accreditation Committee (HAC) and Nick 
Harris (QAA) 

EMLE group work: 

• Project planning 

Chairs: Sara Karlsson (HSV) and Mark Frederiks, Netherlands-Flemish Accreditation 
Organisation (NVAO) 

EURO-AQUAE group work: 

• Project planning 

Chairs: Josep Grifoll, Agency for Quality Assurance in the Catalan University System 
(AQU) and Bruno Curvale, Comite National devaluation (CNE) 

Experts’ group work: 

• Project planning 

Chairs: Staffan Wahlen (HSV), Fiona Crozier (QAA), Tibor Szanto (HAC), Gemma 
Rauret (AQU) and Axel Aerden (NVAO) 

20:00 - 21:30 

Dinner 


One person from each programme and agency, respectively, briefly introduces the programme/agency (three minutes each). 
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FRIDAY 18 MARCH 


TIME 

ACTIVITY 1 

ACTIVITY II 

ACTIVITY III 

ACTIVITY IV 

09:00 - 10:30 

Continued 
CoMundus 
project planning 

Continued 

EMLE 

project planning 

Continued 
EURO-AQUAE 
project planning 

Continued 
project planning 
by the experts 

lO^O - 10:45 

Coffee 




10:45 - 12:00 

CoMundus 
detailed planning 
with experts 

• Introduction of 
experts to the 
group 

• Timetable 

• Project 
development 

EMLE 

detailed planning 
with experts 

• Introduction of 
experts to the 
group 

• Timetable 

• Project 
development 

EURO-AQUAE 
detailed planning 
with experts 

• Introduction of 
experts to the 
group 

• Timetable 

• Project 
development 

(Experts join the 
respective 
programme groups) 

12:00 - 13:00 

General Discussion, all participants 

13:00 - 14:00 

Lunch 

Afternoon 

Departure 
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Appendix V 

Closing conference programme 

Agenda for the TEEP II Final Conference 

Conference venue: Hotel Birger Jarl, Tulegatan 8, Stockholm 
Tel. +46-8-674 1800 

THURSDAY 4 MAY 


TIME 

ACTIVITY 

Morning 

Arrival 

12.00 - 13.00 

Lunch 

13:00 - 13:15 

Welcome by Sigbrit Franke, University Chancellor, National Agency for Higher 
Education (HSV), Sweden 

13:15 - 13:45 

TEEP II and its results, Staffan Wahlen, Senior Advisor (HSV) 

13:45 - 14:45 

TEEP II from the point of view of the programmes and of the expert panels 
Reactions from CoMundus, EMLE and EuroAquae and the three expert panels, 
Chair: Tibor Szanto, Secretary General, Hungarian Accreditation Committee (HAC) 

14:45 - 15:15 

Coffee break 

15:15 - 16:30 

Break-up sessions 
Group 1: 

CoMundus - discussion on 
the programme report 
(experts and programme 
representatives) 

Chairs: Christina Rozsnyai, 
Programme Officer (HAC) 
and Nick Harris, Director 
of Development and 
Enhancement Group, 
Quality Assurance Agency 
for Higher Education (QAA) 

Group 2: 

EMLE - discussion on 
the programme report 
(experts and programme 
representatives) 

Chairs: Staffan Wahlen 
(HSV) and Mark Frederiks, 
Policy Adviser, 
Accreditation Organisation 
of the Netherlands & 
Flanders (NVAO) 

Group 3: 

EUROAQUAE- discussion on 
the programme report 
(experts and programme 
representatives) 

Chairs: Gemma Rauret, 
Director, Agency for Quality 
Assurance in the Catalan 
University System (AQU) 
and Bruno Curvale, Project 
Manager Comite National 
devaluation (CNE) 

16:30 - 17:30 

Feedback and plenary discussion, Chair: Josep Grifoll, Head of Quality Assessment 
Area (AQU) 

19:30 

Dinner 
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FRIDAY 5 MAY 


TIME 

ACTIVITY 

09:00 - 09:15 

Introduction to day 2, Fiona Crozier, Assistant Director (QAA) 

09:15 ~ lOlOO 

Conclusions from the EUA’s European Masters New Evaluation Methodology (EMNEM) 
Project, Presentation by David Crosier, Programme Development Director, European 
University Association (EUA) and by Stefanie Hofmann, Head of Project, Accreditation, 
Certification and Quality Assurance Institute (ACQUIN) and ENQA Vice-President 

lOlOO - 11:00 

Break-up sessions 1 
Group 1: 

CoMundus - discussion on 
the methodological report 
(experts and programme 
representatives) 

Chairs: Christina Rozsnyai 
(HAC) and Nick Harris 
(QAA) 

Group 2: 

EMLE - discussion on 
the methodological report 
(experts and programme 
representatives) 

Chairs: Staffan Wahlen 
(HSV) and Mark Frederilcs 
(NVAO) 

Group 3 : 

EUROAQUAE- discussion on 
the methodological report 
(experts and programme 
representatives) 

Chairs: Gemma Rauret 
(AQU) and Bruno Curvale 
(CNE) 

ll:00 - 11:30 

Coffee break 

11:30 - 12:00 

Feedback including a discussion on a possible follow-up project, 
Chair: Axel Aerden, International Policy Adviser (NVAO) 

12:00 - 12:30 

A view from the European Commission, Ingrid Rigler, 

Directorate General of Education and Culture, European Commission 

12:30 - 13:00 

Final discussion and conclusions, Chair: Staffan Wahlen (HSV) 

13:00 -14:00 

Lunch and departure 


1 It was decided on the spot to mix the groups of the second day, while the Chairs remained the same. 
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ENQA Occasional Paper 9, Methodological report of the Transnational European 
Evaluation Project II (TEEP II), emphasises the importance for joint programmes 
of developing quality assurance and enhancement processes which operate across the 
programme as a whole. The report is an outcome of the Transnational European 
Evaluation Project II (TEEP II) undertaken by ENQA in collaboration with its six 
member agencies between August 2004 and June 2006. It outlines a possible method 
- based on the pilot evaluations by TEEP II of three Erasmus Mundus joint master’s 
programmes - for evaluating similar programmes in the future. 
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